Jul 15 18:58:58.569236 ip-10-0-141-106 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 18:58:58.569247 ip-10-0-141-106 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 18:58:58.569254 ip-10-0-141-106 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 18:58:58.569501 ip-10-0-141-106 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 18:59:08.602435 ip-10-0-141-106 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 18:59:08.602455 ip-10-0-141-106 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8af8b81d03c34cbc9aefae71934f9fad -- Jul 15 19:00:46.224828 ip-10-0-141-106 systemd[1]: Starting Kubernetes Kubelet... Jul 15 19:00:46.680095 ip-10-0-141-106 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:00:46.680095 ip-10-0-141-106 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 19:00:46.680095 ip-10-0-141-106 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:00:46.680095 ip-10-0-141-106 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 19:00:46.680095 ip-10-0-141-106 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:00:46.682982 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.682897 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 19:00:46.690160 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690136 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:00:46.690160 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690156 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:00:46.690160 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690160 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:00:46.690160 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690163 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:00:46.690160 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690166 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:00:46.690160 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690169 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690172 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690175 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690178 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690181 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690183 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690186 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690188 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690191 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690194 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690196 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690199 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690202 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690204 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690207 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690209 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690212 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690214 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690217 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:00:46.690389 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690219 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690222 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690224 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690227 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690238 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690241 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690244 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690246 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690249 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690252 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690255 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690257 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690260 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690262 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690265 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690268 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690271 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690274 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690276 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690279 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690281 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:00:46.690869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690284 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690286 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690289 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690292 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690294 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690297 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690299 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690302 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690305 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690308 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690310 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690312 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690317 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690321 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690324 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690327 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690330 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690333 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690336 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:00:46.691403 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690339 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690342 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690345 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690348 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690351 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690354 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690356 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690360 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690362 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690365 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690367 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690370 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690372 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690376 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690380 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690383 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690386 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690389 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690392 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:00:46.691875 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690396 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690399 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690402 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690405 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690407 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690410 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690413 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690416 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690418 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690421 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690424 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690428 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690431 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690434 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690437 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.690439 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691145 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691152 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691155 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:00:46.692334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691158 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691165 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691169 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691172 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691175 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691178 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691181 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691184 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691187 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691191 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691195 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691198 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691201 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691204 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691208 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691211 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691213 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691216 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691218 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:00:46.692834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691221 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691224 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691226 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691228 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691231 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691234 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691237 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691240 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691242 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691245 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691247 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691250 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691252 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691255 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691258 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691261 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691264 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691266 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691269 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:00:46.693335 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691271 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691274 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691277 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691279 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691282 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691284 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691287 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691290 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691292 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691295 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691298 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691300 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691303 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691305 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691308 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691310 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691313 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691316 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691319 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691321 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:00:46.693835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691323 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691326 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691328 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691331 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691334 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691338 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691340 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691343 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691346 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691348 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691351 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691353 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691356 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691359 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691362 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691365 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691367 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691369 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691372 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691374 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:00:46.694326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691377 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691379 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691382 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691384 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691387 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691390 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691392 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691395 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691397 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691400 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691403 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691405 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691408 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691410 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691413 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691416 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691418 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691421 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691832 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691839 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:00:46.694834 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691842 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691845 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691848 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691851 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691853 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691856 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691858 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691861 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691864 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691866 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691869 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691872 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691874 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691877 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691883 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691886 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691890 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691894 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691897 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691900 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:00:46.695320 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691904 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691907 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691910 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691912 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691916 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691919 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691923 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691925 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691928 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691930 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691933 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691936 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691938 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691941 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691944 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691947 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691949 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691952 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691954 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691957 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:00:46.695827 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691959 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691962 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691965 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691967 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691970 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691972 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691975 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691978 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691980 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691983 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691985 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691988 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691990 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691993 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691996 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.691998 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692001 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692003 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692006 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692008 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:00:46.696315 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692011 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692013 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692016 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692020 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692022 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692025 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692027 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692030 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692032 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692035 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692037 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692039 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692042 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692044 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692047 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692050 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692052 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692055 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692057 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:00:46.696809 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692060 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692064 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692068 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692070 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692073 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692075 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692078 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692080 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692083 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692085 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692088 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692090 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692092 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692095 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692097 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692100 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692103 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692105 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692174 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692181 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692188 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 19:00:46.697272 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692193 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692197 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692200 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692204 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692209 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692213 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692216 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692219 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692222 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692226 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692229 2582 flags.go:64] FLAG: --cgroup-root="" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692231 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692234 2582 flags.go:64] FLAG: --client-ca-file="" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692237 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692243 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692268 2582 flags.go:64] FLAG: --cluster-domain="" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692271 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692275 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692278 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692281 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692285 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692288 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692291 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 19:00:46.697797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692294 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692297 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692300 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692303 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692306 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692309 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692313 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692316 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692319 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692321 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692324 2582 flags.go:64] FLAG: --enable-server="true" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692327 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692332 2582 flags.go:64] FLAG: --event-burst="100" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692335 2582 flags.go:64] FLAG: --event-qps="50" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692339 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692342 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692345 2582 flags.go:64] FLAG: --eviction-hard="" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692349 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692352 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692355 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692357 2582 flags.go:64] FLAG: --eviction-soft="" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692360 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692363 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692367 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692369 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 19:00:46.698345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692372 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692375 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692378 2582 flags.go:64] FLAG: --feature-gates="" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692381 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692384 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692387 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692390 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692392 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692395 2582 flags.go:64] FLAG: --help="false" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692398 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-141-106.ec2.internal" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692401 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692404 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692407 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692411 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692414 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692417 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692419 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692422 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692425 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692427 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692433 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692436 2582 flags.go:64] FLAG: --kube-reserved="" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692439 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692441 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 19:00:46.699003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692444 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692447 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692449 2582 flags.go:64] FLAG: --lock-file="" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692452 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692455 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692458 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692463 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692466 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692469 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692472 2582 flags.go:64] FLAG: --logging-format="text" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692475 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692478 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692480 2582 flags.go:64] FLAG: --manifest-url="" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692483 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692487 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692490 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692494 2582 flags.go:64] FLAG: --max-pods="110" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692496 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692499 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692504 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692507 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692510 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692512 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692515 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692522 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 19:00:46.699819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692525 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692528 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692531 2582 flags.go:64] FLAG: --pod-cidr="" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692550 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692559 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692562 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692565 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692568 2582 flags.go:64] FLAG: --port="10250" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692571 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692574 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c109eb3b46aa9fe8" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692577 2582 flags.go:64] FLAG: --qos-reserved="" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692580 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692583 2582 flags.go:64] FLAG: --register-node="true" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692585 2582 flags.go:64] FLAG: --register-with-taints="" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692589 2582 flags.go:64] FLAG: --registry-burst="10" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692593 2582 flags.go:64] FLAG: --registry-qps="5" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692595 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692598 2582 flags.go:64] FLAG: --reserved-memory="" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692602 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692605 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692608 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692610 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692613 2582 flags.go:64] FLAG: --runonce="false" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692616 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692619 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 19:00:46.700883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692622 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692624 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692629 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692631 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692634 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692637 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692640 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692643 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692645 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692648 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692652 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692655 2582 flags.go:64] FLAG: --system-cgroups="" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692658 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692664 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692666 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692669 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692673 2582 flags.go:64] FLAG: --tls-min-version="" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692675 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692678 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692681 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692684 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692686 2582 flags.go:64] FLAG: --v="2" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692691 2582 flags.go:64] FLAG: --version="false" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692695 2582 flags.go:64] FLAG: --vmodule="" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692698 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 19:00:46.701955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.692701 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692830 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692835 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692838 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692841 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692844 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692848 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692851 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692853 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692857 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692860 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692863 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692866 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692868 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692871 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692873 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692876 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692880 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692883 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692885 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:00:46.703038 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692888 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692890 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692893 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692895 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692898 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692900 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692903 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692905 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692908 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692911 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692914 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692917 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692921 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692924 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692927 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692929 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692932 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692935 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692937 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:00:46.703838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692939 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692942 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692946 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692949 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692953 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692956 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692959 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692962 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692965 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692968 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692972 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692975 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692978 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692981 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692984 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692987 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692990 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692992 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692995 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:00:46.704414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.692997 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693001 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693004 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693007 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693010 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693012 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693015 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693018 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693021 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693023 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693026 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693028 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693031 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693033 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693036 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693039 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693042 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693044 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693047 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:00:46.705020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693050 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693052 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693055 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693057 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693061 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693063 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693066 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693068 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693071 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693073 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693076 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693079 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693081 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693083 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693086 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693088 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693091 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693094 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693097 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693099 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:00:46.705478 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693103 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693106 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.693109 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.693810 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.702076 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.702096 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.702123 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.702132 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702236 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702244 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702250 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702255 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702260 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702264 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:00:46.706314 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702269 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702273 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702280 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702285 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702290 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702294 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702299 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702303 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702308 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702312 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702316 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702320 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702325 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702329 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702333 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702338 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702342 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702346 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702352 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:00:46.706869 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702357 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702361 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702366 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702370 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702375 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702380 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702384 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702388 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702393 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702398 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702402 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702406 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702411 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702415 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702419 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702424 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702428 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702433 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702437 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:00:46.707337 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702441 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702448 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702454 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702459 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702463 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702468 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702472 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702476 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702481 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702485 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702490 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702494 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702498 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702503 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702507 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702512 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702516 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702521 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702526 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:00:46.707931 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702531 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702552 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702557 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702561 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702568 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702573 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702578 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702583 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702587 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702592 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702596 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702601 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702605 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702609 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702614 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702618 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702622 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702626 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702630 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702634 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:00:46.708396 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702638 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702642 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702647 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702652 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702656 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702660 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702664 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702678 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702683 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702687 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702692 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702697 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702701 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702705 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702709 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:00:46.708998 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702714 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.702722 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702942 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702949 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702954 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702959 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702963 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702973 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702977 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702982 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702986 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702990 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.702997 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703003 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:00:46.709364 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703008 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703013 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703017 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703022 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703026 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703031 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703035 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703039 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703043 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703047 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703051 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703062 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703067 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703072 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703076 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703081 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703085 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703089 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703095 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:00:46.709795 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703100 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703104 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703108 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703113 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703117 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703121 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703125 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703129 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703133 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703138 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703142 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703146 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703150 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703154 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703158 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703162 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703167 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703171 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703176 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703180 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:00:46.710263 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703185 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703189 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703193 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703197 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703201 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703215 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703220 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703224 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703228 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703233 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703237 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703240 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703245 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703249 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703253 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703257 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703261 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703265 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703269 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:00:46.710835 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703274 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703278 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703282 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703286 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703290 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703294 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703298 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703303 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703307 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703311 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703315 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703319 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703324 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703328 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703332 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703337 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703349 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703353 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703358 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:00:46.711282 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703374 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703380 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703386 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703390 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703395 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703399 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703403 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703407 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703411 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:46.703415 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.703423 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.703590 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.707640 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.709475 2582 server.go:1019] "Starting client certificate rotation" Jul 15 19:00:46.711860 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.709563 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:00:46.712215 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.709602 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:00:46.712730 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.712716 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:00:46.719876 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.719858 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:00:46.729130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.729112 2582 log.go:25] "Validated CRI v1 runtime API" Jul 15 19:00:46.731484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.731470 2582 log.go:25] "Validated CRI v1 image API" Jul 15 19:00:46.732266 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.732251 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 19:00:46.736031 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.736012 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b1fbd59-bc55-4c6a-aa57-a81cc0d6e0fa:/dev/nvme0n1p4 babbfdc8-bafd-4faa-bee7-e0d3bb0d7be1:/dev/nvme0n1p3] Jul 15 19:00:46.736088 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.736031 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 19:00:46.742064 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.741958 2582 manager.go:216] Machine: {Timestamp:2026-07-15 19:00:46.740644879 +0000 UTC m=+0.379701691 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099097 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20c620ea58a1bfe1ae0953862195d8 SystemUUID:ec20c620-ea58-a1bf-e1ae-0953862195d8 BootID:8af8b81d-03c3-4cbc-9aef-ae71934f9fad Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b3:d5:33:e0:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b3:d5:33:e0:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:1e:07:f4:90:1d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 19:00:46.742064 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.742058 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 19:00:46.742172 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.742160 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 19:00:46.745286 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.745262 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 19:00:46.745413 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.745288 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-106.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 19:00:46.746024 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.746014 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 19:00:46.746055 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.746026 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 19:00:46.746055 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.746034 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:00:46.746785 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.746775 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:00:46.748458 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.748446 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 19:00:46.751947 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.751936 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:00:46.752070 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752061 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 19:00:46.752104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752093 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 15 19:00:46.752104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752102 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 19:00:46.752158 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752116 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 19:00:46.752158 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752125 2582 kubelet.go:398] "Adding apiserver pod source" Jul 15 19:00:46.752158 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752133 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 19:00:46.752510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752492 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:00:46.752600 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.752521 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:00:46.754734 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.754713 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:00:46.754830 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.754763 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 19:00:46.757323 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757307 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 19:00:46.757389 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757338 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 19:00:46.757488 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757477 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 19:00:46.757551 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757492 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 19:00:46.757551 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757498 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 19:00:46.757551 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757503 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 19:00:46.757551 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757511 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 19:00:46.757551 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.757517 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 19:00:46.758127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758118 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 19:00:46.758159 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758128 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 19:00:46.758159 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758135 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 19:00:46.758159 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758140 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 19:00:46.758159 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758148 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 19:00:46.758159 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758156 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 19:00:46.758320 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758181 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 19:00:46.758320 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.758187 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 19:00:46.764742 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.764035 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 19:00:46.764742 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.764090 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 19:00:46.764742 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.764516 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 19:00:46.765019 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.763969 2582 server.go:1268] "Started kubelet" Jul 15 19:00:46.765880 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.765856 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 19:00:46.766036 ip-10-0-141-106 systemd[1]: Started Kubernetes Kubelet. Jul 15 19:00:46.766234 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.766032 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-106.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 19:00:46.766234 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.766113 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 19:00:46.766341 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.766243 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 19:00:46.768313 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.768292 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 15 19:00:46.771884 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.771861 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 19:00:46.773802 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.773780 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 19:00:46.773874 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.773806 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 19:00:46.773925 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.773881 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:46.773970 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.772838 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-106.ec2.internal.18c28bbd1e33f21f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-106.ec2.internal,UID:ip-10-0-141-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-106.ec2.internal,},FirstTimestamp:2026-07-15 19:00:46.763938335 +0000 UTC m=+0.402995152,LastTimestamp:2026-07-15 19:00:46.763938335 +0000 UTC m=+0.402995152,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-106.ec2.internal,}" Jul 15 19:00:46.774488 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774469 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 19:00:46.774617 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774491 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 19:00:46.774671 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774627 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 15 19:00:46.774671 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774635 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 15 19:00:46.774764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774669 2582 factory.go:55] Registering systemd factory Jul 15 19:00:46.774764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774686 2582 factory.go:223] Registration of the systemd container factory successfully Jul 15 19:00:46.774764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774755 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 19:00:46.774917 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774901 2582 factory.go:153] Registering CRI-O factory Jul 15 19:00:46.774973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.774920 2582 factory.go:223] Registration of the crio container factory successfully Jul 15 19:00:46.775025 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.775013 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 19:00:46.775082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.775039 2582 factory.go:103] Registering Raw factory Jul 15 19:00:46.775082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.775054 2582 manager.go:1196] Started watching for new ooms in manager Jul 15 19:00:46.776817 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.776799 2582 manager.go:349] Starting recovery of all containers Jul 15 19:00:46.779754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.779728 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tp4k6" Jul 15 19:00:46.784248 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.784198 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 19:00:46.786114 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.786099 2582 manager.go:354] Recovery completed Jul 15 19:00:46.786364 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.786338 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 19:00:46.786450 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.786421 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 19:00:46.787861 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.787838 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tp4k6" Jul 15 19:00:46.791167 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.791117 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:00:46.793192 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.793175 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:00:46.793260 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.793209 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:00:46.793260 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.793222 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:00:46.793879 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.793854 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 19:00:46.793879 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.793875 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:00:46.794001 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.793877 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 19:00:46.794001 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.793979 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:00:46.795083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.795070 2582 policy_none.go:49] "None policy: Start" Jul 15 19:00:46.795140 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.795087 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 19:00:46.795140 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.795097 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 19:00:46.796271 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.796261 2582 policy_none.go:47] "Start" Jul 15 19:00:46.799251 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.799234 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:00:46.799331 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.799261 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:00:46.799331 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.799275 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:00:46.822150 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.822136 2582 manager.go:337] "Starting Device Plugin manager" Jul 15 19:00:46.822248 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.822165 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 19:00:46.822248 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.822174 2582 server.go:87] "Starting device plugin registration server" Jul 15 19:00:46.822398 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.822384 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 19:00:46.822450 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.822402 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 19:00:46.822550 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.822520 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 19:00:46.822660 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.822643 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 19:00:46.822660 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.822657 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 19:00:46.823556 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.822995 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 19:00:46.823556 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.823046 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:46.923249 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.923221 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:00:46.924289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.924206 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:00:46.924289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.924244 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:00:46.924289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.924261 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:00:46.924454 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.924295 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-106.ec2.internal" Jul 15 19:00:46.932336 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.932288 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-106.ec2.internal" Jul 15 19:00:46.932336 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.932312 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-106.ec2.internal\": node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:46.947122 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.947099 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:46.949030 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.949011 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 19:00:46.949087 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.949052 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 19:00:46.949087 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.949071 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 19:00:46.949157 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:46.949099 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 19:00:46.951465 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:46.951449 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:47.047686 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:47.047655 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:47.049800 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.049776 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal"] Jul 15 19:00:47.049862 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.049839 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:00:47.050736 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.050721 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:00:47.050815 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.050750 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:00:47.050815 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.050765 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:00:47.052080 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.052067 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:00:47.052178 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.052165 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.052214 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.052204 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:00:47.052786 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.052768 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:00:47.052786 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.052788 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:00:47.052911 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.052798 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:00:47.053612 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.053596 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:00:47.053691 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.053621 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:00:47.053691 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.053631 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:00:47.054988 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.054973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.055072 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.055002 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:00:47.055637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.055623 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:00:47.055690 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.055650 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:00:47.055690 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.055665 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:00:47.075059 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:47.075041 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-106.ec2.internal\" not found" node="ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.076590 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.076576 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dd3103017ff04be8f534c63cc011831-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"9dd3103017ff04be8f534c63cc011831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.076650 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.076598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dd3103017ff04be8f534c63cc011831-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"9dd3103017ff04be8f534c63cc011831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.076650 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.076615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98572c5ed91da58bf583a8cefb49b9cd-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"98572c5ed91da58bf583a8cefb49b9cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.079212 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:47.079197 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-106.ec2.internal\" not found" node="ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.148591 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:47.148567 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:47.176817 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.176798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98572c5ed91da58bf583a8cefb49b9cd-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"98572c5ed91da58bf583a8cefb49b9cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.176926 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.176822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dd3103017ff04be8f534c63cc011831-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"9dd3103017ff04be8f534c63cc011831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.176926 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.176843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dd3103017ff04be8f534c63cc011831-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"9dd3103017ff04be8f534c63cc011831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.176926 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.176880 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dd3103017ff04be8f534c63cc011831-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"9dd3103017ff04be8f534c63cc011831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.176926 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.176912 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98572c5ed91da58bf583a8cefb49b9cd-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"98572c5ed91da58bf583a8cefb49b9cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.176926 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.176921 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dd3103017ff04be8f534c63cc011831-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"9dd3103017ff04be8f534c63cc011831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.249153 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:47.249114 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:47.349842 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:47.349805 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:47.377171 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.377140 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Jul 15 19:00:47.382058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:47.382041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:48.694601 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:48.694574 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:48.795099 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:48.795022 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Jul 15 19:00:48.823915 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.823889 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:48.857144 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.857123 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:48.895989 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.895961 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:48.918151 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.918119 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Jul 15 19:00:48.932946 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.932929 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:00:48.933885 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.933872 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:48.952040 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.952023 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 19:00:48.952169 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.952152 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:00:48.952225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.952197 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:00:48.952225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.952213 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:00:48.952319 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:48.952209 2582 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://ac550dae45e2e463fb8974fed4927177-670c361efe1521fc.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.141.106:53662->100.60.188.40:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Jul 15 19:00:48.952319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.952199 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:00:48.996354 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:48.996326 2582 apiserver.go:52] "Watching apiserver" Jul 15 19:00:49.018076 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.018044 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 19:00:49.022823 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.022801 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 19:00:49.023161 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.023139 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-cn2qq","openshift-multus/network-metrics-daemon-fffbr","openshift-network-diagnostics/network-check-target-tbdkm","openshift-network-operator/iptables-alerter-6qt7f","kube-system/konnectivity-agent-4v4sv","kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal","openshift-cluster-node-tuning-operator/tuned-w5mqw","openshift-dns/node-resolver-pc459","openshift-image-registry/node-ca-hw4qx","openshift-multus/multus-additional-cni-plugins-rvzjf","openshift-ovn-kubernetes/ovnkube-node-6jjrx","kube-system/global-pull-secret-syncer-67swt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal"] Jul 15 19:00:49.024591 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.024575 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.026095 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.026075 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:49.026180 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.026160 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:00:49.027181 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.027167 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:49.027230 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.027216 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:00:49.028200 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.028177 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.029302 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.029288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.030478 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.030466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.031653 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.031636 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.032635 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.032620 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.033767 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.033750 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.034900 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.034865 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 18:55:48 +0000 UTC" deadline="2028-01-27 00:28:55.699483076 +0000 UTC" Jul 15 19:00:49.034900 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.034898 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13445h28m6.664587179s" Jul 15 19:00:49.035036 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.034902 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.036054 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.036039 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.036114 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.036097 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:00:49.036830 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.036817 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 19:00:49.037048 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.037035 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zbq2x\"" Jul 15 19:00:49.037115 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.037101 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 19:00:49.037392 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.037375 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 19:00:49.037498 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.037415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.037685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.037666 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 19:00:49.037977 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.037961 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 19:00:49.038027 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.038005 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 19:00:49.038071 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.038005 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m75z5\"" Jul 15 19:00:49.038181 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.038167 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 19:00:49.038593 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.038580 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 19:00:49.038693 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.038680 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 19:00:49.038781 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.038768 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dcldl\"" Jul 15 19:00:49.040564 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.040552 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 19:00:49.040803 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.040788 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 19:00:49.040924 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.040903 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 19:00:49.041020 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.040949 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 19:00:49.041434 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.041336 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 19:00:49.044647 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.044628 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 19:00:49.044744 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.044677 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:00:49.044882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.044863 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 19:00:49.044945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.044880 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 19:00:49.045006 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.044957 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 19:00:49.045006 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.044965 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:00:49.045006 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.044993 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 19:00:49.045186 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045030 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 19:00:49.045186 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045137 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6hxs6\"" Jul 15 19:00:49.045302 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045206 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 19:00:49.045302 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045269 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 19:00:49.045302 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045279 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 19:00:49.045302 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045294 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x8tls\"" Jul 15 19:00:49.045495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045314 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 19:00:49.045495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045400 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j2pz7\"" Jul 15 19:00:49.045495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045457 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 19:00:49.045495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045485 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v6bb8\"" Jul 15 19:00:49.045680 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045619 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h66gz\"" Jul 15 19:00:49.046006 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.045993 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5hg4v\"" Jul 15 19:00:49.050258 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.050226 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:00:49.067752 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.067736 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2bvxh" Jul 15 19:00:49.076434 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.076416 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2bvxh" Jul 15 19:00:49.119664 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.119391 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 19:00:49.124662 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-os-release\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.124754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-netns\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.124754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-etc-kubernetes\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.124754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.124754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-systemd\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.124754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-cni-bin\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124773 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-host\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-tuned\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124846 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-systemd-units\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3aace167-d7c8-4990-af41-4318ad2a5d28-hosts-file\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-system-cni-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-var-lib-kubelet\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.124994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.124989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-systemd\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-slash\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdwq\" (UniqueName: \"kubernetes.io/projected/3aace167-d7c8-4990-af41-4318ad2a5d28-kube-api-access-5jdwq\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125083 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-k8s-cni-cncf-io\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125110 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gs9\" (UniqueName: \"kubernetes.io/projected/cb3da942-a7bc-47c9-9bdd-578463675d5e-kube-api-access-89gs9\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125127 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4465b344-e63b-4844-9941-04810dbc9814-agent-certs\") pod \"konnectivity-agent-4v4sv\" (UID: \"4465b344-e63b-4844-9941-04810dbc9814\") " pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-sys\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-system-cni-dir\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-cnibin\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-cni-multus\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125220 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-daemon-config\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.125252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-cni-netd\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxsz2\" (UniqueName: \"kubernetes.io/projected/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-kube-api-access-jxsz2\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-multus-certs\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-lib-modules\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125363 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6dd216-a749-427d-a800-65fc9bcfc3de-tmp\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-var-lib-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-cni-bin\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92cg2\" (UniqueName: \"kubernetes.io/projected/3952c888-d6f4-4c84-9e4f-363c3865d6b9-kube-api-access-92cg2\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aace167-d7c8-4990-af41-4318ad2a5d28-tmp-dir\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125484 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-dbus\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-etc-selinux\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125525 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht99l\" (UniqueName: \"kubernetes.io/projected/56a49c7c-3673-4907-885c-2f3a39656eef-kube-api-access-ht99l\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125549 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bch\" (UniqueName: \"kubernetes.io/projected/b3bae5c4-d541-4f2f-aaea-e830abbe5212-kube-api-access-h8bch\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-sys-fs\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrv8g\" (UniqueName: \"kubernetes.io/projected/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-kube-api-access-zrv8g\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-socket-dir-parent\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.125666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-run\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-etc-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-log-socket\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3bae5c4-d541-4f2f-aaea-e830abbe5212-serviceca\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125775 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysctl-conf\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-ovn\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovnkube-config\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovn-node-metrics-cert\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-cnibin\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125882 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-conf-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77vm\" (UniqueName: \"kubernetes.io/projected/37857cc3-73dd-4481-bee8-716e4e1646f8-kube-api-access-p77vm\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysconfig\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-kubernetes\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-kubelet\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.125994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-cni-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126016 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3952c888-d6f4-4c84-9e4f-363c3865d6b9-iptables-alerter-script\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3952c888-d6f4-4c84-9e4f-363c3865d6b9-host-slash\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126051 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-node-log\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovnkube-script-lib\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-socket-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-hostroot\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4465b344-e63b-4844-9941-04810dbc9814-konnectivity-ca\") pod \"konnectivity-agent-4v4sv\" (UID: \"4465b344-e63b-4844-9941-04810dbc9814\") " pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysctl-d\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-os-release\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126192 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-run-netns\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126226 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-kubelet-config\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37857cc3-73dd-4481-bee8-716e4e1646f8-cni-binary-copy\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-kubelet\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.126605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-modprobe-d\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.127104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3bae5c4-d541-4f2f-aaea-e830abbe5212-host\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.127104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.127104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qnh\" (UniqueName: \"kubernetes.io/projected/5a6dd216-a749-427d-a800-65fc9bcfc3de-kube-api-access-l7qnh\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.127104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.127104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-env-overrides\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.127104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-registration-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.127104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.126425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-device-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.189482 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.189443 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98572c5ed91da58bf583a8cefb49b9cd.slice/crio-ac1bbcd3ff3902e6955b8b896a8f606f68765e7993f122e675c20d1a6d7acbc2 WatchSource:0}: Error finding container ac1bbcd3ff3902e6955b8b896a8f606f68765e7993f122e675c20d1a6d7acbc2: Status 404 returned error can't find the container with id ac1bbcd3ff3902e6955b8b896a8f606f68765e7993f122e675c20d1a6d7acbc2 Jul 15 19:00:49.196401 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.196384 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:00:49.227322 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:49.227322 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-os-release\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.227479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-run-netns\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.227479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227353 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-kubelet-config\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.227479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37857cc3-73dd-4481-bee8-716e4e1646f8-cni-binary-copy\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.227479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227436 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-os-release\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.227479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227436 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-run-netns\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.227479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227456 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-kubelet-config\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-kubelet\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-modprobe-d\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3bae5c4-d541-4f2f-aaea-e830abbe5212-host\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-kubelet\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227571 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3bae5c4-d541-4f2f-aaea-e830abbe5212-host\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qnh\" (UniqueName: \"kubernetes.io/projected/5a6dd216-a749-427d-a800-65fc9bcfc3de-kube-api-access-l7qnh\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-modprobe-d\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.227642 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-env-overrides\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-registration-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.227697 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret podName:1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34 nodeName:}" failed. No retries permitted until 2026-07-15 19:00:49.727678768 +0000 UTC m=+2.123034717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret") pod "global-pull-secret-syncer-67swt" (UID: "1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-registration-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.227727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227721 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-device-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-os-release\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227763 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-device-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-netns\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-etc-kubernetes\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227822 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-os-release\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227857 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-systemd\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-cni-bin\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-host\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227880 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-netns\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-tuned\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37857cc3-73dd-4481-bee8-716e4e1646f8-cni-binary-copy\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-cni-bin\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-systemd-units\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227913 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-systemd\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-host\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.228464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-etc-kubernetes\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3aace167-d7c8-4990-af41-4318ad2a5d28-hosts-file\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.227999 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-systemd-units\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-system-cni-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-var-lib-kubelet\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3aace167-d7c8-4990-af41-4318ad2a5d28-hosts-file\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-systemd\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228095 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-system-cni-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228134 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-var-lib-kubelet\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-slash\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-env-overrides\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228177 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-systemd\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdwq\" (UniqueName: \"kubernetes.io/projected/3aace167-d7c8-4990-af41-4318ad2a5d28-kube-api-access-5jdwq\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.228213 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:49.229130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228227 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-k8s-cni-cncf-io\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.228317 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs podName:cb3da942-a7bc-47c9-9bdd-578463675d5e nodeName:}" failed. No retries permitted until 2026-07-15 19:00:49.728300008 +0000 UTC m=+2.123655977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs") pod "network-metrics-daemon-fffbr" (UID: "cb3da942-a7bc-47c9-9bdd-578463675d5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-k8s-cni-cncf-io\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-slash\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gs9\" (UniqueName: \"kubernetes.io/projected/cb3da942-a7bc-47c9-9bdd-578463675d5e-kube-api-access-89gs9\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228314 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4465b344-e63b-4844-9941-04810dbc9814-agent-certs\") pod \"konnectivity-agent-4v4sv\" (UID: \"4465b344-e63b-4844-9941-04810dbc9814\") " pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228415 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-sys\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-system-cni-dir\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-cnibin\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-cni-multus\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228496 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-sys\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228530 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-cnibin\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228531 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56a49c7c-3673-4907-885c-2f3a39656eef-system-cni-dir\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.229985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-daemon-config\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-var-lib-cni-multus\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.228939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-daemon-config\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-cni-netd\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxsz2\" (UniqueName: \"kubernetes.io/projected/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-kube-api-access-jxsz2\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-multus-certs\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-lib-modules\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229290 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-cni-netd\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6dd216-a749-427d-a800-65fc9bcfc3de-tmp\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229309 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-host-run-multus-certs\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-var-lib-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-cni-bin\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-var-lib-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229496 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-lib-modules\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92cg2\" (UniqueName: \"kubernetes.io/projected/3952c888-d6f4-4c84-9e4f-363c3865d6b9-kube-api-access-92cg2\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-cni-bin\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aace167-d7c8-4990-af41-4318ad2a5d28-tmp-dir\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-dbus\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.230774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-etc-selinux\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht99l\" (UniqueName: \"kubernetes.io/projected/56a49c7c-3673-4907-885c-2f3a39656eef-kube-api-access-ht99l\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bch\" (UniqueName: \"kubernetes.io/projected/b3bae5c4-d541-4f2f-aaea-e830abbe5212-kube-api-access-h8bch\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229680 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-sys-fs\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrv8g\" (UniqueName: \"kubernetes.io/projected/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-kube-api-access-zrv8g\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-socket-dir-parent\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-run\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-dbus\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-etc-selinux\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-etc-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229805 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-run\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229829 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-log-socket\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3bae5c4-d541-4f2f-aaea-e830abbe5212-serviceca\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysctl-conf\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229943 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-socket-dir-parent\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.231688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229951 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-sys-fs\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229798 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aace167-d7c8-4990-af41-4318ad2a5d28-tmp-dir\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-ovn\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-etc-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.229992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovnkube-config\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230032 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovn-node-metrics-cert\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-cnibin\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230085 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-ovn\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-conf-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230117 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysctl-conf\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77vm\" (UniqueName: \"kubernetes.io/projected/37857cc3-73dd-4481-bee8-716e4e1646f8-kube-api-access-p77vm\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysconfig\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-kubernetes\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-kubelet\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-cni-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3952c888-d6f4-4c84-9e4f-363c3865d6b9-iptables-alerter-script\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3952c888-d6f4-4c84-9e4f-363c3865d6b9-host-slash\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-node-log\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovnkube-script-lib\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-socket-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-hostroot\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4465b344-e63b-4844-9941-04810dbc9814-konnectivity-ca\") pod \"konnectivity-agent-4v4sv\" (UID: \"4465b344-e63b-4844-9941-04810dbc9814\") " pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysctl-d\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230566 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovnkube-config\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3bae5c4-d541-4f2f-aaea-e830abbe5212-serviceca\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-conf-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230647 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysctl-d\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-cnibin\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230054 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-run-openvswitch\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3952c888-d6f4-4c84-9e4f-363c3865d6b9-host-slash\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230907 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-log-socket\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.230967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-sysconfig\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-host-kubelet\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.232916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231177 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3952c888-d6f4-4c84-9e4f-363c3865d6b9-iptables-alerter-script\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231222 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-kubernetes\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231285 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-multus-cni-dir\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231290 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37857cc3-73dd-4481-bee8-716e4e1646f8-hostroot\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-socket-dir\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231339 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-node-log\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6dd216-a749-427d-a800-65fc9bcfc3de-tmp\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.231961 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6dd216-a749-427d-a800-65fc9bcfc3de-etc-tuned\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.232024 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56a49c7c-3673-4907-885c-2f3a39656eef-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.232160 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovnkube-script-lib\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.232196 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4465b344-e63b-4844-9941-04810dbc9814-agent-certs\") pod \"konnectivity-agent-4v4sv\" (UID: \"4465b344-e63b-4844-9941-04810dbc9814\") " pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.233430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.232427 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-ovn-node-metrics-cert\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.235606 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.235590 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4465b344-e63b-4844-9941-04810dbc9814-konnectivity-ca\") pod \"konnectivity-agent-4v4sv\" (UID: \"4465b344-e63b-4844-9941-04810dbc9814\") " pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.240171 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.240153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qnh\" (UniqueName: \"kubernetes.io/projected/5a6dd216-a749-427d-a800-65fc9bcfc3de-kube-api-access-l7qnh\") pod \"tuned-w5mqw\" (UID: \"5a6dd216-a749-427d-a800-65fc9bcfc3de\") " pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.241449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.241431 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxsz2\" (UniqueName: \"kubernetes.io/projected/aa209ff4-1128-4cdb-afd6-aeafafcf0fd2-kube-api-access-jxsz2\") pod \"ovnkube-node-6jjrx\" (UID: \"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.243794 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.243773 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:00:49.243936 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.243799 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:00:49.243936 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.243812 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9wgcq for pod openshift-network-diagnostics/network-check-target-tbdkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:49.243936 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.243881 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq podName:00aa12c5-a721-4c14-8c1c-10c35a916c4d nodeName:}" failed. No retries permitted until 2026-07-15 19:00:49.743865351 +0000 UTC m=+2.139221322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9wgcq" (UniqueName: "kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq") pod "network-check-target-tbdkm" (UID: "00aa12c5-a721-4c14-8c1c-10c35a916c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:49.245608 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.245580 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77vm\" (UniqueName: \"kubernetes.io/projected/37857cc3-73dd-4481-bee8-716e4e1646f8-kube-api-access-p77vm\") pod \"multus-cn2qq\" (UID: \"37857cc3-73dd-4481-bee8-716e4e1646f8\") " pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.246062 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.246028 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht99l\" (UniqueName: \"kubernetes.io/projected/56a49c7c-3673-4907-885c-2f3a39656eef-kube-api-access-ht99l\") pod \"multus-additional-cni-plugins-rvzjf\" (UID: \"56a49c7c-3673-4907-885c-2f3a39656eef\") " pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.246141 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.246031 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrv8g\" (UniqueName: \"kubernetes.io/projected/fcaff9ec-64c7-4a9c-9c5a-2f8145f62640-kube-api-access-zrv8g\") pod \"aws-ebs-csi-driver-node-phbzx\" (UID: \"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.246141 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.246078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdwq\" (UniqueName: \"kubernetes.io/projected/3aace167-d7c8-4990-af41-4318ad2a5d28-kube-api-access-5jdwq\") pod \"node-resolver-pc459\" (UID: \"3aace167-d7c8-4990-af41-4318ad2a5d28\") " pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.246141 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.246128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92cg2\" (UniqueName: \"kubernetes.io/projected/3952c888-d6f4-4c84-9e4f-363c3865d6b9-kube-api-access-92cg2\") pod \"iptables-alerter-6qt7f\" (UID: \"3952c888-d6f4-4c84-9e4f-363c3865d6b9\") " pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.246262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.246203 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gs9\" (UniqueName: \"kubernetes.io/projected/cb3da942-a7bc-47c9-9bdd-578463675d5e-kube-api-access-89gs9\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:49.246978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.246960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bch\" (UniqueName: \"kubernetes.io/projected/b3bae5c4-d541-4f2f-aaea-e830abbe5212-kube-api-access-h8bch\") pod \"node-ca-hw4qx\" (UID: \"b3bae5c4-d541-4f2f-aaea-e830abbe5212\") " pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.361752 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.361672 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cn2qq" Jul 15 19:00:49.370280 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.370236 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37857cc3_73dd_4481_bee8_716e4e1646f8.slice/crio-f8c980fde411eff232c7498c3793a88afd09279b70f08277554b775cab87da86 WatchSource:0}: Error finding container f8c980fde411eff232c7498c3793a88afd09279b70f08277554b775cab87da86: Status 404 returned error can't find the container with id f8c980fde411eff232c7498c3793a88afd09279b70f08277554b775cab87da86 Jul 15 19:00:49.379877 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.379859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6qt7f" Jul 15 19:00:49.386414 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.386394 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3952c888_d6f4_4c84_9e4f_363c3865d6b9.slice/crio-aff2d27445cae1f7905670b7e805248722e7d6873624bf5c6fcb2e68b8dfa8dd WatchSource:0}: Error finding container aff2d27445cae1f7905670b7e805248722e7d6873624bf5c6fcb2e68b8dfa8dd: Status 404 returned error can't find the container with id aff2d27445cae1f7905670b7e805248722e7d6873624bf5c6fcb2e68b8dfa8dd Jul 15 19:00:49.398142 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.398108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:00:49.405516 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.405496 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4465b344_e63b_4844_9941_04810dbc9814.slice/crio-7c37176a0bac49489dd5c9345f3eeb752b0fbfa8319612571e8acd4a9da3d2e4 WatchSource:0}: Error finding container 7c37176a0bac49489dd5c9345f3eeb752b0fbfa8319612571e8acd4a9da3d2e4: Status 404 returned error can't find the container with id 7c37176a0bac49489dd5c9345f3eeb752b0fbfa8319612571e8acd4a9da3d2e4 Jul 15 19:00:49.418776 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.418762 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" Jul 15 19:00:49.431954 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.431937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pc459" Jul 15 19:00:49.439128 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.439104 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aace167_d7c8_4990_af41_4318ad2a5d28.slice/crio-27093b1e529675148f8c9e4a063c55e12a6bd92414ae6e497fbcc8787f8cbbc2 WatchSource:0}: Error finding container 27093b1e529675148f8c9e4a063c55e12a6bd92414ae6e497fbcc8787f8cbbc2: Status 404 returned error can't find the container with id 27093b1e529675148f8c9e4a063c55e12a6bd92414ae6e497fbcc8787f8cbbc2 Jul 15 19:00:49.450878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.450858 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hw4qx" Jul 15 19:00:49.457639 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.457619 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bae5c4_d541_4f2f_aaea_e830abbe5212.slice/crio-7d269d0c26d6e154a86a48a517ecd42d9e957a3679c908ea3c58044442da9d26 WatchSource:0}: Error finding container 7d269d0c26d6e154a86a48a517ecd42d9e957a3679c908ea3c58044442da9d26: Status 404 returned error can't find the container with id 7d269d0c26d6e154a86a48a517ecd42d9e957a3679c908ea3c58044442da9d26 Jul 15 19:00:49.471509 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.471489 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" Jul 15 19:00:49.478460 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.478444 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:00:49.478657 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.478638 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a49c7c_3673_4907_885c_2f3a39656eef.slice/crio-d4f8e06f3c85fd67592fa53c43caa535e059084ffd49fdf3df419365121134fa WatchSource:0}: Error finding container d4f8e06f3c85fd67592fa53c43caa535e059084ffd49fdf3df419365121134fa: Status 404 returned error can't find the container with id d4f8e06f3c85fd67592fa53c43caa535e059084ffd49fdf3df419365121134fa Jul 15 19:00:49.484996 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.484975 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa209ff4_1128_4cdb_afd6_aeafafcf0fd2.slice/crio-dcdd475d9de7ee64bfdbc084cc5be385e9bf57d434af0b7342efce176492f163 WatchSource:0}: Error finding container dcdd475d9de7ee64bfdbc084cc5be385e9bf57d434af0b7342efce176492f163: Status 404 returned error can't find the container with id dcdd475d9de7ee64bfdbc084cc5be385e9bf57d434af0b7342efce176492f163 Jul 15 19:00:49.496332 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.496315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" Jul 15 19:00:49.503815 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:00:49.503794 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcaff9ec_64c7_4a9c_9c5a_2f8145f62640.slice/crio-e081f9bc4109407648f8c63f546e56538ae41572f2f2d75d8b8adb93885ded23 WatchSource:0}: Error finding container e081f9bc4109407648f8c63f546e56538ae41572f2f2d75d8b8adb93885ded23: Status 404 returned error can't find the container with id e081f9bc4109407648f8c63f546e56538ae41572f2f2d75d8b8adb93885ded23 Jul 15 19:00:49.733325 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.733298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:49.733519 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.733365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:49.733519 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.733478 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:49.733646 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.733579 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret podName:1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34 nodeName:}" failed. No retries permitted until 2026-07-15 19:00:50.733556862 +0000 UTC m=+3.128912813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret") pod "global-pull-secret-syncer-67swt" (UID: "1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:49.733646 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.733477 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:49.733646 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.733621 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs podName:cb3da942-a7bc-47c9-9bdd-578463675d5e nodeName:}" failed. No retries permitted until 2026-07-15 19:00:50.733610162 +0000 UTC m=+3.128966111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs") pod "network-metrics-daemon-fffbr" (UID: "cb3da942-a7bc-47c9-9bdd-578463675d5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:49.834442 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:49.834410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:49.834622 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.834545 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:00:49.834622 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.834563 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:00:49.834622 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.834574 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9wgcq for pod openshift-network-diagnostics/network-check-target-tbdkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:49.834771 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:49.834636 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq podName:00aa12c5-a721-4c14-8c1c-10c35a916c4d nodeName:}" failed. No retries permitted until 2026-07-15 19:00:50.83461593 +0000 UTC m=+3.229971879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9wgcq" (UniqueName: "kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq") pod "network-check-target-tbdkm" (UID: "00aa12c5-a721-4c14-8c1c-10c35a916c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:50.078430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.078315 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 18:55:49 +0000 UTC" deadline="2028-01-03 16:13:56.781955692 +0000 UTC" Jul 15 19:00:50.078430 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.078389 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12885h13m6.703571531s" Jul 15 19:00:50.196318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.196282 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:50.196490 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.196418 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:00:50.203104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.203051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4v4sv" event={"ID":"4465b344-e63b-4844-9941-04810dbc9814","Type":"ContainerStarted","Data":"7c37176a0bac49489dd5c9345f3eeb752b0fbfa8319612571e8acd4a9da3d2e4"} Jul 15 19:00:50.204886 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.204861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" event={"ID":"98572c5ed91da58bf583a8cefb49b9cd","Type":"ContainerStarted","Data":"ac1bbcd3ff3902e6955b8b896a8f606f68765e7993f122e675c20d1a6d7acbc2"} Jul 15 19:00:50.206744 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.206719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" event={"ID":"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640","Type":"ContainerStarted","Data":"e081f9bc4109407648f8c63f546e56538ae41572f2f2d75d8b8adb93885ded23"} Jul 15 19:00:50.209261 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.209222 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerStarted","Data":"d4f8e06f3c85fd67592fa53c43caa535e059084ffd49fdf3df419365121134fa"} Jul 15 19:00:50.213497 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.213473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hw4qx" event={"ID":"b3bae5c4-d541-4f2f-aaea-e830abbe5212","Type":"ContainerStarted","Data":"7d269d0c26d6e154a86a48a517ecd42d9e957a3679c908ea3c58044442da9d26"} Jul 15 19:00:50.218977 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.218683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pc459" event={"ID":"3aace167-d7c8-4990-af41-4318ad2a5d28","Type":"ContainerStarted","Data":"27093b1e529675148f8c9e4a063c55e12a6bd92414ae6e497fbcc8787f8cbbc2"} Jul 15 19:00:50.223176 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.223078 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" event={"ID":"5a6dd216-a749-427d-a800-65fc9bcfc3de","Type":"ContainerStarted","Data":"7fb2eb2d7128f498d92e1d960f6e4bb6208349c8cc24dcf4ed503d88b008867e"} Jul 15 19:00:50.227171 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.227101 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6qt7f" event={"ID":"3952c888-d6f4-4c84-9e4f-363c3865d6b9","Type":"ContainerStarted","Data":"aff2d27445cae1f7905670b7e805248722e7d6873624bf5c6fcb2e68b8dfa8dd"} Jul 15 19:00:50.228841 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.228755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cn2qq" event={"ID":"37857cc3-73dd-4481-bee8-716e4e1646f8","Type":"ContainerStarted","Data":"f8c980fde411eff232c7498c3793a88afd09279b70f08277554b775cab87da86"} Jul 15 19:00:50.231088 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.231066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"dcdd475d9de7ee64bfdbc084cc5be385e9bf57d434af0b7342efce176492f163"} Jul 15 19:00:50.280700 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.280670 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:50.740881 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.740844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:50.741127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.740943 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:50.741127 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.741068 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:50.741127 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.741122 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret podName:1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34 nodeName:}" failed. No retries permitted until 2026-07-15 19:00:52.741104569 +0000 UTC m=+5.136460534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret") pod "global-pull-secret-syncer-67swt" (UID: "1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:50.741633 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.741526 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:50.741633 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.741577 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs podName:cb3da942-a7bc-47c9-9bdd-578463675d5e nodeName:}" failed. No retries permitted until 2026-07-15 19:00:52.741560982 +0000 UTC m=+5.136916934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs") pod "network-metrics-daemon-fffbr" (UID: "cb3da942-a7bc-47c9-9bdd-578463675d5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:50.841595 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:50.841563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:50.841778 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.841728 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:00:50.841778 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.841749 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:00:50.841778 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.841761 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9wgcq for pod openshift-network-diagnostics/network-check-target-tbdkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:50.841930 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:50.841816 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq podName:00aa12c5-a721-4c14-8c1c-10c35a916c4d nodeName:}" failed. No retries permitted until 2026-07-15 19:00:52.841798177 +0000 UTC m=+5.237154138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9wgcq" (UniqueName: "kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq") pod "network-check-target-tbdkm" (UID: "00aa12c5-a721-4c14-8c1c-10c35a916c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:51.078654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:51.078536 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 18:55:49 +0000 UTC" deadline="2028-01-04 17:27:19.863564775 +0000 UTC" Jul 15 19:00:51.078654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:51.078576 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12910h26m28.784996411s" Jul 15 19:00:51.194567 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:51.194044 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:51.194567 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:51.194186 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:00:51.194567 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:51.194341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:51.194567 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:51.194456 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:00:51.242037 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:51.242007 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:51.342113 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:51.341851 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:51.610573 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:51.610171 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:00:52.195554 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:52.195527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:52.196063 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.195658 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:00:52.758126 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:52.758093 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:52.758324 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:52.758144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:52.758324 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.758280 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:52.758424 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.758341 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs podName:cb3da942-a7bc-47c9-9bdd-578463675d5e nodeName:}" failed. No retries permitted until 2026-07-15 19:00:56.758324063 +0000 UTC m=+9.153680013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs") pod "network-metrics-daemon-fffbr" (UID: "cb3da942-a7bc-47c9-9bdd-578463675d5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:52.758806 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.758718 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:52.758806 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.758768 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret podName:1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34 nodeName:}" failed. No retries permitted until 2026-07-15 19:00:56.758752987 +0000 UTC m=+9.154108939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret") pod "global-pull-secret-syncer-67swt" (UID: "1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:52.859308 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:52.859266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:52.859550 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.859528 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:00:52.859550 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.859552 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:00:52.859692 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.859566 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9wgcq for pod openshift-network-diagnostics/network-check-target-tbdkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:52.859692 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:52.859644 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq podName:00aa12c5-a721-4c14-8c1c-10c35a916c4d nodeName:}" failed. No retries permitted until 2026-07-15 19:00:56.859611644 +0000 UTC m=+9.254967614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9wgcq" (UniqueName: "kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq") pod "network-check-target-tbdkm" (UID: "00aa12c5-a721-4c14-8c1c-10c35a916c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:53.193489 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:53.193409 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:53.193651 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:53.193546 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:00:53.193651 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:53.193637 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:53.193799 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:53.193757 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:00:54.195271 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:54.194986 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:54.195271 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:54.195130 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:00:55.193025 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:55.192992 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:55.193191 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:55.193124 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:00:55.193510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:55.193485 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:55.193619 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:55.193601 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:00:56.193597 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:56.193565 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:56.194023 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.193711 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:00:56.794902 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:56.794863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:56.795063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:56.794968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:56.795063 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.794992 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:56.795063 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.795062 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs podName:cb3da942-a7bc-47c9-9bdd-578463675d5e nodeName:}" failed. No retries permitted until 2026-07-15 19:01:04.795040661 +0000 UTC m=+17.190396609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs") pod "network-metrics-daemon-fffbr" (UID: "cb3da942-a7bc-47c9-9bdd-578463675d5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:00:56.795258 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.795074 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:56.795258 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.795124 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret podName:1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:04.79510862 +0000 UTC m=+17.190464575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret") pod "global-pull-secret-syncer-67swt" (UID: "1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:00:56.895621 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:56.895543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:56.895778 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.895711 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:00:56.895778 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.895735 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:00:56.895778 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.895750 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9wgcq for pod openshift-network-diagnostics/network-check-target-tbdkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:56.895937 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:56.895841 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq podName:00aa12c5-a721-4c14-8c1c-10c35a916c4d nodeName:}" failed. No retries permitted until 2026-07-15 19:01:04.895821535 +0000 UTC m=+17.291177494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9wgcq" (UniqueName: "kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq") pod "network-check-target-tbdkm" (UID: "00aa12c5-a721-4c14-8c1c-10c35a916c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:00:57.193188 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:57.193093 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:57.193365 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:57.193254 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:00:57.193794 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:57.193632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:57.193794 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:57.193755 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:00:58.194621 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:58.194585 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:00:58.195048 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:58.194706 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:00:59.193901 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:59.193869 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:00:59.194073 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:00:59.193869 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:00:59.194073 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:59.193996 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:00:59.194190 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:00:59.194068 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:00.193155 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:00.193121 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:00.193569 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:00.193280 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:01.193419 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:01.193384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:01.193866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:01.193384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:01.193866 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:01.193535 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:01.193866 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:01.193589 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:02.193650 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:02.193619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:02.194109 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:02.193750 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:03.193739 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:03.193709 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:03.194100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:03.193709 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:03.194100 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:03.193810 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:03.194100 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:03.193885 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:04.193733 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:04.193643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:04.193893 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.193791 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:04.858446 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:04.858267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:04.858639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:04.858466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:04.858639 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.858422 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:04.858639 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.858564 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:04.858639 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.858566 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret podName:1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:20.85854923 +0000 UTC m=+33.253905181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret") pod "global-pull-secret-syncer-67swt" (UID: "1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:04.858639 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.858619 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs podName:cb3da942-a7bc-47c9-9bdd-578463675d5e nodeName:}" failed. No retries permitted until 2026-07-15 19:01:20.858603051 +0000 UTC m=+33.253959001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs") pod "network-metrics-daemon-fffbr" (UID: "cb3da942-a7bc-47c9-9bdd-578463675d5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:04.959198 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:04.959164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:04.959390 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.959342 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:04.959390 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.959365 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:04.959390 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.959379 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9wgcq for pod openshift-network-diagnostics/network-check-target-tbdkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:04.959515 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:04.959443 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq podName:00aa12c5-a721-4c14-8c1c-10c35a916c4d nodeName:}" failed. No retries permitted until 2026-07-15 19:01:20.959423088 +0000 UTC m=+33.354779041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9wgcq" (UniqueName: "kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq") pod "network-check-target-tbdkm" (UID: "00aa12c5-a721-4c14-8c1c-10c35a916c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:05.193704 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:05.193624 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:05.193867 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:05.193749 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:05.193867 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:05.193801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:05.194231 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:05.193912 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:06.193935 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:06.193900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:06.194404 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:06.194038 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:07.193908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:07.193866 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:07.194078 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:07.193871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:07.194078 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:07.194001 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:07.194515 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:07.194130 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:08.194654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:08.194619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:08.195135 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:08.194744 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:08.825170 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:08.825148 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd3103017ff04be8f534c63cc011831.slice/crio-5a6e66a0b358ef59e3dbcbb576fb8f5bb30303b1f557f36d1b93628302b0bd56 WatchSource:0}: Error finding container 5a6e66a0b358ef59e3dbcbb576fb8f5bb30303b1f557f36d1b93628302b0bd56: Status 404 returned error can't find the container with id 5a6e66a0b358ef59e3dbcbb576fb8f5bb30303b1f557f36d1b93628302b0bd56 Jul 15 19:01:09.193448 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.193379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:09.193448 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.193410 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:09.193625 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:09.193469 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:09.193625 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:09.193583 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:09.271940 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.271908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" event={"ID":"5a6dd216-a749-427d-a800-65fc9bcfc3de","Type":"ContainerStarted","Data":"c56c5a6808b05531584d8f1df8676f2d3b3b5f5be4da128c67bf96171efc9b48"} Jul 15 19:01:09.273518 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.273492 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cn2qq" event={"ID":"37857cc3-73dd-4481-bee8-716e4e1646f8","Type":"ContainerStarted","Data":"b15ec94318ca9264e41391885aa501320b2e815d75932a73c050c6544825e18e"} Jul 15 19:01:09.274878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.274851 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" event={"ID":"98572c5ed91da58bf583a8cefb49b9cd","Type":"ContainerStarted","Data":"180e034771aa0c33a688c9edea6d400158d304a8e782a6df6702862870d1b8db"} Jul 15 19:01:09.278661 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.278631 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"9dd3103017ff04be8f534c63cc011831","Type":"ContainerStarted","Data":"5a6e66a0b358ef59e3dbcbb576fb8f5bb30303b1f557f36d1b93628302b0bd56"} Jul 15 19:01:09.298258 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.298112 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w5mqw" podStartSLOduration=2.228007689 podStartE2EDuration="21.298092771s" totalImagesPullingTime="19.070085082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:09.297559543 +0000 UTC m=+21.692915514" watchObservedRunningTime="2026-07-15 19:01:09.298092771 +0000 UTC m=+21.693448742" Jul 15 19:01:09.361110 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:09.359064 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" podStartSLOduration=21.359037199 podStartE2EDuration="21.359037199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:09.318495473 +0000 UTC m=+21.713851444" watchObservedRunningTime="2026-07-15 19:01:09.359037199 +0000 UTC m=+21.754393170" Jul 15 19:01:10.194040 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.193848 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:10.194176 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:10.194094 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:10.282343 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.282133 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" event={"ID":"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640","Type":"ContainerStarted","Data":"e35528107fc18f9ae76a57050b335f5697216afe1d2f14ee5138f2251b49e738"} Jul 15 19:01:10.284018 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.283985 2582 generic.go:356] "Generic (PLEG): container finished" podID="56a49c7c-3673-4907-885c-2f3a39656eef" containerID="6e2d7556564cae75e6ee4b7d5c4bd847ae8f044ade9d393f5004c27a672d0d79" exitCode=0 Jul 15 19:01:10.284170 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.284083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerDied","Data":"6e2d7556564cae75e6ee4b7d5c4bd847ae8f044ade9d393f5004c27a672d0d79"} Jul 15 19:01:10.285809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.285761 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hw4qx" event={"ID":"b3bae5c4-d541-4f2f-aaea-e830abbe5212","Type":"ContainerStarted","Data":"8c2694cb95fb61a359972c8dde3138d60bbd103159fce4c93d197a1766b551d6"} Jul 15 19:01:10.287339 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.287316 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pc459" event={"ID":"3aace167-d7c8-4990-af41-4318ad2a5d28","Type":"ContainerStarted","Data":"1c1d853ef43b15ab047d4b39bbf00942d41b8f85295e14ecb6649bdb8ff18628"} Jul 15 19:01:10.289227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.289206 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:01:10.289599 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.289574 2582 generic.go:356] "Generic (PLEG): container finished" podID="aa209ff4-1128-4cdb-afd6-aeafafcf0fd2" containerID="478a8c38bce626b23a67364cf2e08e72f731706990a8dfe074c39eae4cdf9b2d" exitCode=1 Jul 15 19:01:10.289714 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.289648 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerDied","Data":"478a8c38bce626b23a67364cf2e08e72f731706990a8dfe074c39eae4cdf9b2d"} Jul 15 19:01:10.289790 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.289718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"7b0f53156f59c597ba13efd70c24aef473fa68b4a1a175a3c8bd4dbf7f9b0c90"} Jul 15 19:01:10.291382 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.291327 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4v4sv" event={"ID":"4465b344-e63b-4844-9941-04810dbc9814","Type":"ContainerStarted","Data":"765ad2bccdabf9c48a07c8b3b4ca3831f363825ce2ea7949f21e02c42a557bd5"} Jul 15 19:01:10.311579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.311539 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cn2qq" podStartSLOduration=2.849216788 podStartE2EDuration="22.311525786s" totalImagesPullingTime="19.462308998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:09.363131777 +0000 UTC m=+21.758487753" watchObservedRunningTime="2026-07-15 19:01:10.311525786 +0000 UTC m=+22.706881775" Jul 15 19:01:10.327359 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.327316 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4v4sv" podStartSLOduration=2.9167052509999998 podStartE2EDuration="22.327301726s" totalImagesPullingTime="19.410596475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:10.326695405 +0000 UTC m=+22.722051376" watchObservedRunningTime="2026-07-15 19:01:10.327301726 +0000 UTC m=+22.722657718" Jul 15 19:01:10.342855 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.342820 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pc459" podStartSLOduration=2.961813047 podStartE2EDuration="22.342808271s" totalImagesPullingTime="19.380995224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:10.342573725 +0000 UTC m=+22.737929718" watchObservedRunningTime="2026-07-15 19:01:10.342808271 +0000 UTC m=+22.738164240" Jul 15 19:01:10.866050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:10.866005 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 19:01:11.077537 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.077354 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T19:01:10.866026519Z","UUID":"e2ed16ec-6278-4adc-8d9f-775d22c0de01","Handler":null,"Name":"","Endpoint":""} Jul 15 19:01:11.078883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.078865 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 19:01:11.078995 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.078898 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 19:01:11.193521 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.193487 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:11.193751 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.193487 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:11.193751 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:11.193599 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:11.193751 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:11.193671 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:11.295344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.295307 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" event={"ID":"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640","Type":"ContainerStarted","Data":"1fa612ffc212dd8eb8698e452f8d5c5c2e6aa14846adaca2e97cdb412c9f5e62"} Jul 15 19:01:11.296908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.296875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6qt7f" event={"ID":"3952c888-d6f4-4c84-9e4f-363c3865d6b9","Type":"ContainerStarted","Data":"bb0d4188e1b524fdaf8c45955a6aa9c11a12e941606d2f7a781a39295bb65504"} Jul 15 19:01:11.313077 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.313028 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hw4qx" podStartSLOduration=4.274863589 podStartE2EDuration="23.313014915s" totalImagesPullingTime="19.038151326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:10.361052454 +0000 UTC m=+22.756408424" watchObservedRunningTime="2026-07-15 19:01:11.313014915 +0000 UTC m=+23.708370884" Jul 15 19:01:11.313736 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:11.313708 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6qt7f" podStartSLOduration=3.940952715 podStartE2EDuration="23.313697419s" totalImagesPullingTime="19.372744704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:11.313149918 +0000 UTC m=+23.708505869" watchObservedRunningTime="2026-07-15 19:01:11.313697419 +0000 UTC m=+23.709053397" Jul 15 19:01:12.193709 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:12.193688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:12.193831 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:12.193809 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:12.300935 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:12.300916 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:01:12.301321 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:12.301267 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"971b95a4d4e3e4590e358ed2d1ece941b0534811881de2e0840662cdb5862a60"} Jul 15 19:01:13.128190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.128156 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:13.189392 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.189365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:13.193230 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.193213 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:13.193337 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:13.193320 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:13.193385 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.193368 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:13.193465 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:13.193450 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:13.305803 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.305653 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:01:13.306164 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.306145 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"4b964c0bbab03e60043eea8f9cb810b3bf74a58795f549c7f86a55bca843af40"} Jul 15 19:01:13.306230 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.306217 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"5073ef8983c0cbd489201a6b5ad0908a205d8c917c4bc5b8fd42500bf2762bbe"} Jul 15 19:01:13.306313 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.306282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"e050b3faeae1a6a40e25cf801d4a88f5227ccadd9fcdd1d67f179c059e188495"} Jul 15 19:01:13.307602 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.307579 2582 generic.go:356] "Generic (PLEG): container finished" podID="9dd3103017ff04be8f534c63cc011831" containerID="5dd45101cb5c84a907154038497adba5d22152186c7d7a9e97f1952e12b0ac36" exitCode=0 Jul 15 19:01:13.307692 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.307617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"9dd3103017ff04be8f534c63cc011831","Type":"ContainerDied","Data":"5dd45101cb5c84a907154038497adba5d22152186c7d7a9e97f1952e12b0ac36"} Jul 15 19:01:13.711875 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.711798 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:01:13.712322 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:13.712303 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:01:14.196541 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:14.196514 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:14.196716 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:14.196637 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:14.312857 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:14.312776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"9dd3103017ff04be8f534c63cc011831","Type":"ContainerStarted","Data":"9e2032e05ecae93556f69b477313fb42c6a68da2cbc4104d5c9db3d7ff1075a6"} Jul 15 19:01:14.313444 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:14.313062 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:01:14.313603 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:14.313584 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4v4sv" Jul 15 19:01:14.331283 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:14.331216 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" podStartSLOduration=26.331198873 podStartE2EDuration="26.331198873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:14.330495488 +0000 UTC m=+26.725851495" watchObservedRunningTime="2026-07-15 19:01:14.331198873 +0000 UTC m=+26.726554845" Jul 15 19:01:14.380507 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:14.380472 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:14.485088 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:14.485049 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:15.193217 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.193138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:15.193400 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.193151 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:15.193400 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:15.193263 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:15.193400 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:15.193360 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:15.318275 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.318233 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:01:15.318813 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.318604 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"ed7ce6761ffa3f1164c3fb8e95a02acba7166206daaa6dcd2a0e2a8ef9747a53"} Jul 15 19:01:15.321103 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.321069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" event={"ID":"fcaff9ec-64c7-4a9c-9c5a-2f8145f62640","Type":"ContainerStarted","Data":"198df985b44c11850e57c874f17d3abe36e1f76cc97ef901ced65e1636352f4c"} Jul 15 19:01:15.340116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.340066 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phbzx" podStartSLOduration=3.145273633 podStartE2EDuration="27.340047744s" totalImagesPullingTime="24.194774111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:15.340005482 +0000 UTC m=+27.735361451" watchObservedRunningTime="2026-07-15 19:01:15.340047744 +0000 UTC m=+27.735403715" Jul 15 19:01:15.416719 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.416682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:15.515839 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:15.515803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:16.199144 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:16.199114 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:16.199320 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:16.199259 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:17.193657 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:17.193577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:17.194100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:17.193577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:17.194100 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:17.193706 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:17.194100 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:17.193804 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:18.194807 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.194592 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:18.195458 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:18.194806 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:18.330099 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.330069 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:01:18.330482 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.330461 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"31ed8487e49197bcb5d16f7744cf0495b72af43df727b5c698b8b746a05219e7"} Jul 15 19:01:18.330723 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.330697 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:01:18.330863 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.330847 2582 scope.go:117] "RemoveContainer" containerID="478a8c38bce626b23a67364cf2e08e72f731706990a8dfe074c39eae4cdf9b2d" Jul 15 19:01:18.332284 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.332261 2582 generic.go:356] "Generic (PLEG): container finished" podID="56a49c7c-3673-4907-885c-2f3a39656eef" containerID="f4b79c508c9980020ebbf06b5c971e21046580d2f78d9c4d8dec771364e8e993" exitCode=0 Jul 15 19:01:18.332360 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.332299 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerDied","Data":"f4b79c508c9980020ebbf06b5c971e21046580d2f78d9c4d8dec771364e8e993"} Jul 15 19:01:18.346371 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.346354 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:01:18.947002 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:18.946971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:19.006919 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.006888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:19.193292 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.193208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:19.193292 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.193257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:19.193486 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:19.193343 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:19.193486 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:19.193455 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:19.336626 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.336595 2582 generic.go:356] "Generic (PLEG): container finished" podID="56a49c7c-3673-4907-885c-2f3a39656eef" containerID="aeeb3a01f6a69f6f972d9fcb04b1064ca34ea2b91aa9ff7ecec45e286551457a" exitCode=0 Jul 15 19:01:19.337023 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.336630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerDied","Data":"aeeb3a01f6a69f6f972d9fcb04b1064ca34ea2b91aa9ff7ecec45e286551457a"} Jul 15 19:01:19.340863 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.340846 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:01:19.341211 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.341183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" event={"ID":"aa209ff4-1128-4cdb-afd6-aeafafcf0fd2","Type":"ContainerStarted","Data":"37a4c0676a0690cf80a7182f6cc451d4a82fb50b648eb92ad8742d9f21f91715"} Jul 15 19:01:19.341440 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.341418 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:01:19.341530 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.341490 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:01:19.355753 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.355735 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:01:19.393389 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.393345 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" podStartSLOduration=8.899342806 podStartE2EDuration="31.393330564s" totalImagesPullingTime="22.493987758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:19.393145522 +0000 UTC m=+31.788501518" watchObservedRunningTime="2026-07-15 19:01:19.393330564 +0000 UTC m=+31.788686534" Jul 15 19:01:19.431466 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.431437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:19.887466 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.887433 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tbdkm"] Jul 15 19:01:19.887603 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.887583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:19.887697 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:19.887674 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:19.892431 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.892405 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-67swt"] Jul 15 19:01:19.892553 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.892519 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:19.892631 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:19.892609 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:19.893014 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.892993 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fffbr"] Jul 15 19:01:19.893116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:19.893102 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:19.893216 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:19.893199 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:20.112528 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:20.112507 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pc459_3aace167-d7c8-4990-af41-4318ad2a5d28/dns-node-resolver/0.log" Jul 15 19:01:20.345408 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:20.345318 2582 generic.go:356] "Generic (PLEG): container finished" podID="56a49c7c-3673-4907-885c-2f3a39656eef" containerID="6e40217f505e5eb2e80754ed86d146e3ace067082a992ee5b200c3e966d30234" exitCode=0 Jul 15 19:01:20.345802 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:20.345410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerDied","Data":"6e40217f505e5eb2e80754ed86d146e3ace067082a992ee5b200c3e966d30234"} Jul 15 19:01:20.883168 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:20.883140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:20.883332 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:20.883177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:20.883385 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.883363 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:20.883385 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.883374 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:20.883445 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.883425 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs podName:cb3da942-a7bc-47c9-9bdd-578463675d5e nodeName:}" failed. No retries permitted until 2026-07-15 19:01:52.883410251 +0000 UTC m=+65.278766200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs") pod "network-metrics-daemon-fffbr" (UID: "cb3da942-a7bc-47c9-9bdd-578463675d5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:20.883445 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.883438 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret podName:1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:52.883432123 +0000 UTC m=+65.278788070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret") pod "global-pull-secret-syncer-67swt" (UID: "1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:20.906073 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:20.906050 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hw4qx_b3bae5c4-d541-4f2f-aaea-e830abbe5212/node-ca/0.log" Jul 15 19:01:20.983622 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:20.983586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:20.983776 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.983758 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:20.983828 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.983782 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:20.983828 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.983792 2582 projected.go:196] Error preparing data for projected volume kube-api-access-9wgcq for pod openshift-network-diagnostics/network-check-target-tbdkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:20.983901 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:20.983853 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq podName:00aa12c5-a721-4c14-8c1c-10c35a916c4d nodeName:}" failed. No retries permitted until 2026-07-15 19:01:52.983834481 +0000 UTC m=+65.379190443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9wgcq" (UniqueName: "kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq") pod "network-check-target-tbdkm" (UID: "00aa12c5-a721-4c14-8c1c-10c35a916c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:22.105796 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:22.105763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:22.193733 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:22.193702 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:22.193733 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:22.193728 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:22.193946 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:22.193703 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:22.193946 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:22.193807 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-67swt" podUID="1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34" Jul 15 19:01:22.193946 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:22.193903 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbdkm" podUID="00aa12c5-a721-4c14-8c1c-10c35a916c4d" Jul 15 19:01:22.194088 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:22.193998 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fffbr" podUID="cb3da942-a7bc-47c9-9bdd-578463675d5e" Jul 15 19:01:22.205339 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:22.205313 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:22.782171 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:22.782080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:23.896858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.896782 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeReady" Jul 15 19:01:23.897495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.896941 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 19:01:23.931698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.931556 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bddd547bb-4sddm"] Jul 15 19:01:23.933774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.933752 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:23.936336 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.936315 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 19:01:23.936453 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.936343 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 19:01:23.936453 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.936360 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 19:01:23.936658 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.936641 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z9lvq\"" Jul 15 19:01:23.942586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.942572 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bljkh"] Jul 15 19:01:23.943421 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.943401 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 19:01:23.944713 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.944689 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:23.947151 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.947085 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 19:01:23.947151 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.947107 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 19:01:23.947151 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.947112 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 19:01:23.947882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.947860 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jgfdg\"" Jul 15 19:01:23.948345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.948325 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8shls"] Jul 15 19:01:23.950464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.950272 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:23.952400 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.951615 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bddd547bb-4sddm"] Jul 15 19:01:23.953856 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.953265 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 19:01:23.953856 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.953367 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tn2w9\"" Jul 15 19:01:23.953856 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.953273 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 19:01:23.953856 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.953711 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 19:01:23.953856 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.953777 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 19:01:23.954338 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.954319 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bljkh"] Jul 15 19:01:23.964932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.964912 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8shls"] Jul 15 19:01:23.968042 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.968022 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mw954"] Jul 15 19:01:23.970103 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.970084 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mw954" Jul 15 19:01:23.972459 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.972424 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 19:01:23.972580 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.972558 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 19:01:23.972673 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.972655 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgrg9\"" Jul 15 19:01:23.981360 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:23.981343 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mw954"] Jul 15 19:01:24.077171 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.077136 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:24.103396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f62bae5e-1f0d-4d93-92a3-e272e8d9353f-cert\") pod \"ingress-canary-bljkh\" (UID: \"f62bae5e-1f0d-4d93-92a3-e272e8d9353f\") " pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:24.103562 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlcdc\" (UniqueName: \"kubernetes.io/projected/36635b29-e242-40ea-89c8-75dffb3c7324-kube-api-access-hlcdc\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.103562 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-installation-pull-secrets\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.103562 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58ff17fe-ed94-43f3-9798-816ae901a86f-data-volume\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.103562 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxrg\" (UniqueName: \"kubernetes.io/projected/f62bae5e-1f0d-4d93-92a3-e272e8d9353f-kube-api-access-5mxrg\") pod \"ingress-canary-bljkh\" (UID: \"f62bae5e-1f0d-4d93-92a3-e272e8d9353f\") " pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:24.103562 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103541 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-trusted-ca\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103567 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36635b29-e242-40ea-89c8-75dffb3c7324-config-volume\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103626 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36635b29-e242-40ea-89c8-75dffb3c7324-tmp-dir\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-bound-sa-token\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6r4\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-kube-api-access-fj6r4\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-certificates\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7km7\" (UniqueName: \"kubernetes.io/projected/58ff17fe-ed94-43f3-9798-816ae901a86f-kube-api-access-q7km7\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.103801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.104082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103804 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58ff17fe-ed94-43f3-9798-816ae901a86f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.104082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ba2f16-961e-4e01-ba43-933a25bcae99-ca-trust-extracted\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.104082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58ff17fe-ed94-43f3-9798-816ae901a86f-crio-socket\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.104082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.104082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.103971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-image-registry-private-configuration\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.178933 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.178828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:24.193809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.193762 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:24.193809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.193776 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:24.194008 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.193993 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:24.196564 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.196445 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:01:24.196564 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.196471 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:01:24.196564 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.196445 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j7rjq\"" Jul 15 19:01:24.196564 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.196455 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:01:24.196844 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.196661 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7hbwd\"" Jul 15 19:01:24.196844 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.196797 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:01:24.205137 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205155 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-image-registry-private-configuration\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f62bae5e-1f0d-4d93-92a3-e272e8d9353f-cert\") pod \"ingress-canary-bljkh\" (UID: \"f62bae5e-1f0d-4d93-92a3-e272e8d9353f\") " pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlcdc\" (UniqueName: \"kubernetes.io/projected/36635b29-e242-40ea-89c8-75dffb3c7324-kube-api-access-hlcdc\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-installation-pull-secrets\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58ff17fe-ed94-43f3-9798-816ae901a86f-data-volume\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.205233 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205283 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxrg\" (UniqueName: \"kubernetes.io/projected/f62bae5e-1f0d-4d93-92a3-e272e8d9353f-kube-api-access-5mxrg\") pod \"ingress-canary-bljkh\" (UID: \"f62bae5e-1f0d-4d93-92a3-e272e8d9353f\") " pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:24.205269 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.205318 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls podName:58ff17fe-ed94-43f3-9798-816ae901a86f nodeName:}" failed. No retries permitted until 2026-07-15 19:01:24.705295229 +0000 UTC m=+37.100651193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8shls" (UID: "58ff17fe-ed94-43f3-9798-816ae901a86f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205341 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-trusted-ca\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36635b29-e242-40ea-89c8-75dffb3c7324-config-volume\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36635b29-e242-40ea-89c8-75dffb3c7324-tmp-dir\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-bound-sa-token\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6r4\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-kube-api-access-fj6r4\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205540 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-certificates\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7km7\" (UniqueName: \"kubernetes.io/projected/58ff17fe-ed94-43f3-9798-816ae901a86f-kube-api-access-q7km7\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58ff17fe-ed94-43f3-9798-816ae901a86f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ba2f16-961e-4e01-ba43-933a25bcae99-ca-trust-extracted\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.205717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.205673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58ff17fe-ed94-43f3-9798-816ae901a86f-crio-socket\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.206300 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.206023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58ff17fe-ed94-43f3-9798-816ae901a86f-data-volume\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.206300 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.206042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36635b29-e242-40ea-89c8-75dffb3c7324-config-volume\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.206300 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.206073 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58ff17fe-ed94-43f3-9798-816ae901a86f-crio-socket\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.206300 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.206145 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:01:24.206300 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.206157 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bddd547bb-4sddm: secret "image-registry-tls" not found Jul 15 19:01:24.206300 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.206194 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls podName:30ba2f16-961e-4e01-ba43-933a25bcae99 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:24.706181139 +0000 UTC m=+37.101537087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls") pod "image-registry-bddd547bb-4sddm" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99") : secret "image-registry-tls" not found Jul 15 19:01:24.206933 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.206630 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-certificates\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.206933 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.206754 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36635b29-e242-40ea-89c8-75dffb3c7324-tmp-dir\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.206933 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.206918 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58ff17fe-ed94-43f3-9798-816ae901a86f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.207118 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.206953 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:01:24.207118 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.207005 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls podName:36635b29-e242-40ea-89c8-75dffb3c7324 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:24.70698898 +0000 UTC m=+37.102344935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls") pod "dns-default-mw954" (UID: "36635b29-e242-40ea-89c8-75dffb3c7324") : secret "dns-default-metrics-tls" not found Jul 15 19:01:24.207118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.207016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ba2f16-961e-4e01-ba43-933a25bcae99-ca-trust-extracted\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.207118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.207042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-trusted-ca\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.210330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.210304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-image-registry-private-configuration\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.210330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.210326 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-installation-pull-secrets\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.210490 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.210334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f62bae5e-1f0d-4d93-92a3-e272e8d9353f-cert\") pod \"ingress-canary-bljkh\" (UID: \"f62bae5e-1f0d-4d93-92a3-e272e8d9353f\") " pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:24.216883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.216859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxrg\" (UniqueName: \"kubernetes.io/projected/f62bae5e-1f0d-4d93-92a3-e272e8d9353f-kube-api-access-5mxrg\") pod \"ingress-canary-bljkh\" (UID: \"f62bae5e-1f0d-4d93-92a3-e272e8d9353f\") " pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:24.217210 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.217165 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6r4\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-kube-api-access-fj6r4\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.217530 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.217365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlcdc\" (UniqueName: \"kubernetes.io/projected/36635b29-e242-40ea-89c8-75dffb3c7324-kube-api-access-hlcdc\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.217730 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.217657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7km7\" (UniqueName: \"kubernetes.io/projected/58ff17fe-ed94-43f3-9798-816ae901a86f-kube-api-access-q7km7\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.218379 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.218358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-bound-sa-token\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.256116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.256088 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bljkh" Jul 15 19:01:24.404063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.403843 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bljkh"] Jul 15 19:01:24.407523 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:24.407493 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62bae5e_1f0d_4d93_92a3_e272e8d9353f.slice/crio-dd78930c7ebcaec3a339faca8ed339d1da833a2a85ac49833ece89c744a110b7 WatchSource:0}: Error finding container dd78930c7ebcaec3a339faca8ed339d1da833a2a85ac49833ece89c744a110b7: Status 404 returned error can't find the container with id dd78930c7ebcaec3a339faca8ed339d1da833a2a85ac49833ece89c744a110b7 Jul 15 19:01:24.441513 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.441376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:24.542019 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.541948 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:24.681576 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.681498 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:24.710335 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.710233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:24.710335 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.710295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.710387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.710391 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.710407 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bddd547bb-4sddm: secret "image-registry-tls" not found Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.710459 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.710480 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.710460 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls podName:30ba2f16-961e-4e01-ba43-933a25bcae99 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:25.710440949 +0000 UTC m=+38.105796898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls") pod "image-registry-bddd547bb-4sddm" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99") : secret "image-registry-tls" not found Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.710531 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls podName:58ff17fe-ed94-43f3-9798-816ae901a86f nodeName:}" failed. No retries permitted until 2026-07-15 19:01:25.710514094 +0000 UTC m=+38.105870051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8shls" (UID: "58ff17fe-ed94-43f3-9798-816ae901a86f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:01:24.710545 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:24.710549 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls podName:36635b29-e242-40ea-89c8-75dffb3c7324 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:25.710538236 +0000 UTC m=+38.105894185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls") pod "dns-default-mw954" (UID: "36635b29-e242-40ea-89c8-75dffb3c7324") : secret "dns-default-metrics-tls" not found Jul 15 19:01:24.782878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.782836 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:24.913429 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:24.913397 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.014019 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.013987 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.359268 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.359167 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bljkh" event={"ID":"f62bae5e-1f0d-4d93-92a3-e272e8d9353f","Type":"ContainerStarted","Data":"dd78930c7ebcaec3a339faca8ed339d1da833a2a85ac49833ece89c744a110b7"} Jul 15 19:01:25.387298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.387233 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.498797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.498762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.589951 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.589912 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.675175 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.675097 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.719544 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.719503 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:25.719709 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:25.719688 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:01:25.719783 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:25.719768 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls podName:58ff17fe-ed94-43f3-9798-816ae901a86f nodeName:}" failed. No retries permitted until 2026-07-15 19:01:27.719746335 +0000 UTC m=+40.115102288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8shls" (UID: "58ff17fe-ed94-43f3-9798-816ae901a86f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:01:25.719783 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.719696 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:25.719894 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:25.719783 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:01:25.719894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.719835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:25.719894 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:25.719856 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls podName:36635b29-e242-40ea-89c8-75dffb3c7324 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:27.719839333 +0000 UTC m=+40.115195283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls") pod "dns-default-mw954" (UID: "36635b29-e242-40ea-89c8-75dffb3c7324") : secret "dns-default-metrics-tls" not found Jul 15 19:01:25.720044 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:25.719908 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:01:25.720044 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:25.719920 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bddd547bb-4sddm: secret "image-registry-tls" not found Jul 15 19:01:25.720044 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:25.719963 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls podName:30ba2f16-961e-4e01-ba43-933a25bcae99 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:27.719950193 +0000 UTC m=+40.115306158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls") pod "image-registry-bddd547bb-4sddm" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99") : secret "image-registry-tls" not found Jul 15 19:01:25.770606 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.770575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.872728 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.872693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:25.969606 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:25.969567 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:26.661944 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:26.661897 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:26.723676 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:26.723642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:26.986630 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:26.986602 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-5ldk8"] Jul 15 19:01:27.010883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.010851 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-5ldk8"] Jul 15 19:01:27.011026 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.010944 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.013865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.013843 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 19:01:27.013988 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.013846 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 19:01:27.013988 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.013916 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 19:01:27.014140 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.014128 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 19:01:27.014919 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.014902 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gpkvk\"" Jul 15 19:01:27.103577 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.103546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:27.130564 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.130537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4f8ba81-89dd-4e5e-ac99-9705979496b6-signing-key\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.130670 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.130586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4f8ba81-89dd-4e5e-ac99-9705979496b6-signing-cabundle\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.130736 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.130717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9c6\" (UniqueName: \"kubernetes.io/projected/c4f8ba81-89dd-4e5e-ac99-9705979496b6-kube-api-access-nd9c6\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.181637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.181606 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:27.232069 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.232041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9c6\" (UniqueName: \"kubernetes.io/projected/c4f8ba81-89dd-4e5e-ac99-9705979496b6-kube-api-access-nd9c6\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.232188 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.232083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4f8ba81-89dd-4e5e-ac99-9705979496b6-signing-key\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.232188 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.232114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4f8ba81-89dd-4e5e-ac99-9705979496b6-signing-cabundle\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.232821 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.232799 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4f8ba81-89dd-4e5e-ac99-9705979496b6-signing-cabundle\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.235733 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.235709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4f8ba81-89dd-4e5e-ac99-9705979496b6-signing-key\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.240550 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.240532 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9c6\" (UniqueName: \"kubernetes.io/projected/c4f8ba81-89dd-4e5e-ac99-9705979496b6-kube-api-access-nd9c6\") pod \"service-ca-79f68b9f4f-5ldk8\" (UID: \"c4f8ba81-89dd-4e5e-ac99-9705979496b6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.320013 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.319987 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" Jul 15 19:01:27.369802 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.369677 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerStarted","Data":"5c0b645f05a229ac484a4496e0c7fd0405572620bffe07e298d4fec8a471288a"} Jul 15 19:01:27.524756 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.524685 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-5ldk8"] Jul 15 19:01:27.528597 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:27.528565 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f8ba81_89dd_4e5e_ac99_9705979496b6.slice/crio-75f591f3580273b3aebcd23180a1470c6cd2a52cc30e3d398f5d2a32ee7e6532 WatchSource:0}: Error finding container 75f591f3580273b3aebcd23180a1470c6cd2a52cc30e3d398f5d2a32ee7e6532: Status 404 returned error can't find the container with id 75f591f3580273b3aebcd23180a1470c6cd2a52cc30e3d398f5d2a32ee7e6532 Jul 15 19:01:27.737876 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.737844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:27.738060 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.737895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:27.738060 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:27.737928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:27.738060 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:27.738026 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:01:27.738060 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:27.738047 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:01:27.738060 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:27.738052 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:01:27.738277 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:27.738068 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bddd547bb-4sddm: secret "image-registry-tls" not found Jul 15 19:01:27.738277 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:27.738104 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls podName:36635b29-e242-40ea-89c8-75dffb3c7324 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:31.738083343 +0000 UTC m=+44.133439295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls") pod "dns-default-mw954" (UID: "36635b29-e242-40ea-89c8-75dffb3c7324") : secret "dns-default-metrics-tls" not found Jul 15 19:01:27.738277 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:27.738124 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls podName:58ff17fe-ed94-43f3-9798-816ae901a86f nodeName:}" failed. No retries permitted until 2026-07-15 19:01:31.738114778 +0000 UTC m=+44.133470732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8shls" (UID: "58ff17fe-ed94-43f3-9798-816ae901a86f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:01:27.738277 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:27.738140 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls podName:30ba2f16-961e-4e01-ba43-933a25bcae99 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:31.738131899 +0000 UTC m=+44.133487867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls") pod "image-registry-bddd547bb-4sddm" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99") : secret "image-registry-tls" not found Jul 15 19:01:28.373037 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:28.373004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" event={"ID":"c4f8ba81-89dd-4e5e-ac99-9705979496b6","Type":"ContainerStarted","Data":"75f591f3580273b3aebcd23180a1470c6cd2a52cc30e3d398f5d2a32ee7e6532"} Jul 15 19:01:28.376235 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:28.376210 2582 generic.go:356] "Generic (PLEG): container finished" podID="56a49c7c-3673-4907-885c-2f3a39656eef" containerID="5c0b645f05a229ac484a4496e0c7fd0405572620bffe07e298d4fec8a471288a" exitCode=0 Jul 15 19:01:28.376376 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:28.376287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerDied","Data":"5c0b645f05a229ac484a4496e0c7fd0405572620bffe07e298d4fec8a471288a"} Jul 15 19:01:28.723195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:28.723167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:28.794065 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:28.793990 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:29.382946 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:29.382911 2582 generic.go:356] "Generic (PLEG): container finished" podID="56a49c7c-3673-4907-885c-2f3a39656eef" containerID="aaad60acf4e4c6a18c4322df5c8dcae33133a98ecda15555622a2f91a724c6f0" exitCode=0 Jul 15 19:01:29.383562 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:29.382994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerDied","Data":"aaad60acf4e4c6a18c4322df5c8dcae33133a98ecda15555622a2f91a724c6f0"} Jul 15 19:01:29.765196 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:29.765139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:30.387858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:30.387818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bljkh" event={"ID":"f62bae5e-1f0d-4d93-92a3-e272e8d9353f","Type":"ContainerStarted","Data":"65ffcca9b3fd530dcf4bf4ec052b2e1a1391f66276dea6ac8818b205c7aa150f"} Jul 15 19:01:30.391206 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:30.391178 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" event={"ID":"56a49c7c-3673-4907-885c-2f3a39656eef","Type":"ContainerStarted","Data":"d2eb972811f6f6f738f0a645c248a1e33e49eeaf84ec09489bd262d8e2b1cd16"} Jul 15 19:01:30.407295 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:30.407234 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bljkh" podStartSLOduration=1.606683663 podStartE2EDuration="7.407219869s" totalImagesPullingTime="5.800536206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:30.405992921 +0000 UTC m=+42.801348890" watchObservedRunningTime="2026-07-15 19:01:30.407219869 +0000 UTC m=+42.802575838" Jul 15 19:01:30.431911 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:30.431867 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rvzjf" podStartSLOduration=7.249077692 podStartE2EDuration="42.431856149s" totalImagesPullingTime="35.182778457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:30.430160835 +0000 UTC m=+42.825516805" watchObservedRunningTime="2026-07-15 19:01:30.431856149 +0000 UTC m=+42.827212121" Jul 15 19:01:31.770005 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:31.769966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:31.770005 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:31.770011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:31.770103 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:31.770105 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:31.770133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:31.770167 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls podName:36635b29-e242-40ea-89c8-75dffb3c7324 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:39.770151586 +0000 UTC m=+52.165507534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls") pod "dns-default-mw954" (UID: "36635b29-e242-40ea-89c8-75dffb3c7324") : secret "dns-default-metrics-tls" not found Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:31.770114 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bddd547bb-4sddm: secret "image-registry-tls" not found Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:31.770202 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:31.770228 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls podName:30ba2f16-961e-4e01-ba43-933a25bcae99 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:39.770212827 +0000 UTC m=+52.165568778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls") pod "image-registry-bddd547bb-4sddm" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99") : secret "image-registry-tls" not found Jul 15 19:01:31.770512 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:01:31.770284 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls podName:58ff17fe-ed94-43f3-9798-816ae901a86f nodeName:}" failed. No retries permitted until 2026-07-15 19:01:39.770277061 +0000 UTC m=+52.165633009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8shls" (UID: "58ff17fe-ed94-43f3-9798-816ae901a86f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:01:32.396715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:32.396685 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" event={"ID":"c4f8ba81-89dd-4e5e-ac99-9705979496b6","Type":"ContainerStarted","Data":"6a5e24b6d57cd24932947911dbae61e6acc2b16df2628fa9da3de98634092e12"} Jul 15 19:01:32.412842 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:32.412790 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-5ldk8" podStartSLOduration=2.344481828 podStartE2EDuration="6.412762688s" totalImagesPullingTime="4.06828086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:32.412179707 +0000 UTC m=+44.807535676" watchObservedRunningTime="2026-07-15 19:01:32.412762688 +0000 UTC m=+44.808118662" Jul 15 19:01:33.017063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:33.017030 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:33.134307 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:33.134266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:33.274188 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:33.274074 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:33.351451 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:33.351420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:33.421390 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:33.421357 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:33.723200 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:33.723165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:34.722360 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:34.722318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:34.915593 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:34.915546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:35.075225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:35.075143 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:35.171715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:35.171664 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:35.315289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:35.315256 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:35.514947 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:35.514915 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:36.109893 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:36.109862 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:36.518147 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:36.518109 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:36.708661 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:36.708624 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:36.839397 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:36.839318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:36.940637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:36.940605 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:37.040162 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:37.040120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:37.314978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:37.314932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:37.710795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:37.710701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:37.821466 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:37.821434 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:39.840917 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.840877 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:39.841414 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.840928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:39.841414 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.840965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:39.844343 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.844304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36635b29-e242-40ea-89c8-75dffb3c7324-metrics-tls\") pod \"dns-default-mw954\" (UID: \"36635b29-e242-40ea-89c8-75dffb3c7324\") " pod="openshift-dns/dns-default-mw954" Jul 15 19:01:39.844343 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.844304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58ff17fe-ed94-43f3-9798-816ae901a86f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8shls\" (UID: \"58ff17fe-ed94-43f3-9798-816ae901a86f\") " pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:39.844495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.844341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"image-registry-bddd547bb-4sddm\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:39.846185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.846172 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:39.863589 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.863569 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8shls" Jul 15 19:01:39.879723 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:39.879703 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mw954" Jul 15 19:01:40.024976 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.024878 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bddd547bb-4sddm"] Jul 15 19:01:40.050199 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.050161 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8shls"] Jul 15 19:01:40.053382 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:40.053357 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ff17fe_ed94_43f3_9798_816ae901a86f.slice/crio-82b630ec76d963a8e05b79863f9d1bfecb563437709bad3d76cc7d0c9f2738c3 WatchSource:0}: Error finding container 82b630ec76d963a8e05b79863f9d1bfecb563437709bad3d76cc7d0c9f2738c3: Status 404 returned error can't find the container with id 82b630ec76d963a8e05b79863f9d1bfecb563437709bad3d76cc7d0c9f2738c3 Jul 15 19:01:40.063967 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.063944 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mw954"] Jul 15 19:01:40.067767 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:40.067745 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36635b29_e242_40ea_89c8_75dffb3c7324.slice/crio-77b573d196f92f6e4cec642dce9e1afbf7283950705e9a6795d4def53234dbf2 WatchSource:0}: Error finding container 77b573d196f92f6e4cec642dce9e1afbf7283950705e9a6795d4def53234dbf2: Status 404 returned error can't find the container with id 77b573d196f92f6e4cec642dce9e1afbf7283950705e9a6795d4def53234dbf2 Jul 15 19:01:40.419484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.419387 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8shls" event={"ID":"58ff17fe-ed94-43f3-9798-816ae901a86f","Type":"ContainerStarted","Data":"82b630ec76d963a8e05b79863f9d1bfecb563437709bad3d76cc7d0c9f2738c3"} Jul 15 19:01:40.421163 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.421104 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" event={"ID":"30ba2f16-961e-4e01-ba43-933a25bcae99","Type":"ContainerStarted","Data":"1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b"} Jul 15 19:01:40.421163 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.421158 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" event={"ID":"30ba2f16-961e-4e01-ba43-933a25bcae99","Type":"ContainerStarted","Data":"a750fdd29bf14e15e4c8797fe453aad754f5406559c735c43b2796454191538f"} Jul 15 19:01:40.421386 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.421186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:01:40.422465 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.422428 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mw954" event={"ID":"36635b29-e242-40ea-89c8-75dffb3c7324","Type":"ContainerStarted","Data":"77b573d196f92f6e4cec642dce9e1afbf7283950705e9a6795d4def53234dbf2"} Jul 15 19:01:40.441013 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:40.440970 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" podStartSLOduration=29.440955829 podStartE2EDuration="29.440955829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:40.440818642 +0000 UTC m=+52.836174613" watchObservedRunningTime="2026-07-15 19:01:40.440955829 +0000 UTC m=+52.836311798" Jul 15 19:01:41.006157 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:41.006091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:41.107005 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:41.106923 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:41.176656 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:41.176583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:41.512817 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:41.512756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:41.730277 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:41.730220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:41.915987 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:41.915834 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:42.110840 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:42.110802 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:43.119625 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:43.119590 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:43.317125 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:43.317084 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:43.513255 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:43.513202 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:43.714545 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:43.714512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:46.442356 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:46.442320 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mw954" event={"ID":"36635b29-e242-40ea-89c8-75dffb3c7324","Type":"ContainerStarted","Data":"9d9518cfe45551dcbe4bea2efb161363ae989153f3ccbf70806bf3b5c2f6be40"} Jul 15 19:01:46.442800 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:46.442466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mw954" Jul 15 19:01:46.442800 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:46.442511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mw954" event={"ID":"36635b29-e242-40ea-89c8-75dffb3c7324","Type":"ContainerStarted","Data":"f316a2acc74ab69804c85b7863fbaa567f5cddbb76b1a4c64de677352f57a4ff"} Jul 15 19:01:46.443673 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:46.443651 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8shls" event={"ID":"58ff17fe-ed94-43f3-9798-816ae901a86f","Type":"ContainerStarted","Data":"4784600e784fd0ed73f75398642a73eca353ffff29c131ec6ad9ea9ccdbf835a"} Jul 15 19:01:46.460925 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:46.460876 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mw954" podStartSLOduration=17.846671303 podStartE2EDuration="23.460862962s" totalImagesPullingTime="5.614191659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:46.45930166 +0000 UTC m=+58.854657627" watchObservedRunningTime="2026-07-15 19:01:46.460862962 +0000 UTC m=+58.856218929" Jul 15 19:01:48.751018 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:48.750981 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:48.848842 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:48.848795 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:48.947347 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:48.947266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:49.126230 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:49.126150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:50.086758 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:50.086716 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:50.186040 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:50.186003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:50.458859 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:50.458828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8shls" event={"ID":"58ff17fe-ed94-43f3-9798-816ae901a86f","Type":"ContainerStarted","Data":"bfac6f46d5d94a823d97ffe0a6b7e92d7f608215fefe5705cd17ff7b20024f46"} Jul 15 19:01:51.359622 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:51.359597 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jjrx" Jul 15 19:01:52.950200 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:52.950162 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:52.950632 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:52.950210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:52.952891 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:52.952862 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:01:52.953058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:52.952945 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:01:52.963827 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:52.963797 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3da942-a7bc-47c9-9bdd-578463675d5e-metrics-certs\") pod \"network-metrics-daemon-fffbr\" (UID: \"cb3da942-a7bc-47c9-9bdd-578463675d5e\") " pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:52.963943 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:52.963810 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34-original-pull-secret\") pod \"global-pull-secret-syncer-67swt\" (UID: \"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34\") " pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:53.008748 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.008716 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7hbwd\"" Jul 15 19:01:53.017195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.017160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fffbr" Jul 15 19:01:53.036679 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.036643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-67swt" Jul 15 19:01:53.050762 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.050730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:53.053519 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.053493 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:01:53.063965 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.063941 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:01:53.074729 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.074693 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgcq\" (UniqueName: \"kubernetes.io/projected/00aa12c5-a721-4c14-8c1c-10c35a916c4d-kube-api-access-9wgcq\") pod \"network-check-target-tbdkm\" (UID: \"00aa12c5-a721-4c14-8c1c-10c35a916c4d\") " pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:53.333652 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.333619 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j7rjq\"" Jul 15 19:01:53.341586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.341557 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:01:53.514584 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.514542 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7fmfw"] Jul 15 19:01:53.521354 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.521313 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9"] Jul 15 19:01:53.521680 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.521654 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.524865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.524831 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q"] Jul 15 19:01:53.524989 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.524931 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.525949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.525672 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 15 19:01:53.525949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.525846 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hq99z\"" Jul 15 19:01:53.525949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.525628 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 15 19:01:53.527185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.527170 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.532071 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.531302 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 19:01:53.532071 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.531300 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 19:01:53.532071 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.531508 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 19:01:53.532071 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.531601 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 19:01:53.532071 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.531831 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 19:01:53.532839 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.532107 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 19:01:53.532839 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.532268 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 19:01:53.532947 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.532504 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 19:01:53.533781 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.533763 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7fmfw"] Jul 15 19:01:53.539435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.539408 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:53.544468 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.544433 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9"] Jul 15 19:01:53.566773 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.566745 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q"] Jul 15 19:01:53.568083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.567920 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bddd547bb-4sddm"] Jul 15 19:01:53.611638 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.611561 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-94b8f45fb-n2shh"] Jul 15 19:01:53.615969 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.615940 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.630998 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.630973 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-94b8f45fb-n2shh"] Jul 15 19:01:53.653320 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.653289 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:53.655132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655098 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-hub\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.655271 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655136 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz69g\" (UniqueName: \"kubernetes.io/projected/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-kube-api-access-rz69g\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.655271 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655175 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8b8f985-f6a5-4b0b-9588-663c9aab1550-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7fmfw\" (UID: \"c8b8f985-f6a5-4b0b-9588-663c9aab1550\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.655363 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.655363 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655332 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmxn4\" (UniqueName: \"kubernetes.io/projected/1febcb52-5110-4f8c-89b0-95338089a9bd-kube-api-access-vmxn4\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.655448 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-ca\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.655448 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655398 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1febcb52-5110-4f8c-89b0-95338089a9bd-klusterlet-config\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.655448 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1febcb52-5110-4f8c-89b0-95338089a9bd-tmp\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.655592 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.655592 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.655592 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.655557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8b8f985-f6a5-4b0b-9588-663c9aab1550-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7fmfw\" (UID: \"c8b8f985-f6a5-4b0b-9588-663c9aab1550\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.739318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.739288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:53.756196 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756163 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1febcb52-5110-4f8c-89b0-95338089a9bd-klusterlet-config\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c40cce77-bb87-40d5-82b0-b8c7708ec552-installation-pull-secrets\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1febcb52-5110-4f8c-89b0-95338089a9bd-tmp\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756292 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c40cce77-bb87-40d5-82b0-b8c7708ec552-ca-trust-extracted\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756342 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c40cce77-bb87-40d5-82b0-b8c7708ec552-image-registry-private-configuration\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8b8f985-f6a5-4b0b-9588-663c9aab1550-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7fmfw\" (UID: \"c8b8f985-f6a5-4b0b-9588-663c9aab1550\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.756425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c40cce77-bb87-40d5-82b0-b8c7708ec552-registry-certificates\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-hub\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-registry-tls\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz69g\" (UniqueName: \"kubernetes.io/projected/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-kube-api-access-rz69g\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8b8f985-f6a5-4b0b-9588-663c9aab1550-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7fmfw\" (UID: \"c8b8f985-f6a5-4b0b-9588-663c9aab1550\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c40cce77-bb87-40d5-82b0-b8c7708ec552-trusted-ca\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756631 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1febcb52-5110-4f8c-89b0-95338089a9bd-tmp\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5h6g\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-kube-api-access-m5h6g\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-bound-sa-token\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmxn4\" (UniqueName: \"kubernetes.io/projected/1febcb52-5110-4f8c-89b0-95338089a9bd-kube-api-access-vmxn4\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.756843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.756737 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-ca\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.757410 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.757077 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.757410 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.757320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8b8f985-f6a5-4b0b-9588-663c9aab1550-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7fmfw\" (UID: \"c8b8f985-f6a5-4b0b-9588-663c9aab1550\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.759589 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.759558 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.759828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.759593 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-ca\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.759828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.759644 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-hub\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.760110 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.760087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8b8f985-f6a5-4b0b-9588-663c9aab1550-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7fmfw\" (UID: \"c8b8f985-f6a5-4b0b-9588-663c9aab1550\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.760182 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.760168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.760259 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.760199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1febcb52-5110-4f8c-89b0-95338089a9bd-klusterlet-config\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.766744 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.766722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmxn4\" (UniqueName: \"kubernetes.io/projected/1febcb52-5110-4f8c-89b0-95338089a9bd-kube-api-access-vmxn4\") pod \"klusterlet-addon-workmgr-57f7cc866-6s54q\" (UID: \"1febcb52-5110-4f8c-89b0-95338089a9bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.766867 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.766741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz69g\" (UniqueName: \"kubernetes.io/projected/8cdc045a-4eb7-4e71-86d8-57bb3b7632ca-kube-api-access-rz69g\") pod \"cluster-proxy-proxy-agent-64bd56c7b-c9tm9\" (UID: \"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.839355 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.839286 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c40cce77-bb87-40d5-82b0-b8c7708ec552-image-registry-private-configuration\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c40cce77-bb87-40d5-82b0-b8c7708ec552-registry-certificates\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-registry-tls\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857628 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c40cce77-bb87-40d5-82b0-b8c7708ec552-trusted-ca\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5h6g\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-kube-api-access-m5h6g\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857690 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-bound-sa-token\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c40cce77-bb87-40d5-82b0-b8c7708ec552-installation-pull-secrets\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.857768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c40cce77-bb87-40d5-82b0-b8c7708ec552-ca-trust-extracted\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.858835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c40cce77-bb87-40d5-82b0-b8c7708ec552-registry-certificates\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.859698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.859432 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c40cce77-bb87-40d5-82b0-b8c7708ec552-ca-trust-extracted\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.860950 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.860906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c40cce77-bb87-40d5-82b0-b8c7708ec552-trusted-ca\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.863728 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.862960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-registry-tls\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.864514 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.864263 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" Jul 15 19:01:53.864731 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.864686 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c40cce77-bb87-40d5-82b0-b8c7708ec552-image-registry-private-configuration\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.864910 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.864874 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c40cce77-bb87-40d5-82b0-b8c7708ec552-installation-pull-secrets\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.871568 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.871507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-bound-sa-token\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.871738 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.871713 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5h6g\" (UniqueName: \"kubernetes.io/projected/c40cce77-bb87-40d5-82b0-b8c7708ec552-kube-api-access-m5h6g\") pod \"image-registry-94b8f45fb-n2shh\" (UID: \"c40cce77-bb87-40d5-82b0-b8c7708ec552\") " pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.871850 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.871818 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:01:53.922818 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.922786 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fffbr"] Jul 15 19:01:53.927768 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.926787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:53.928794 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:53.928736 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3da942_a7bc_47c9_9bdd_578463675d5e.slice/crio-dde927857a3ee098a63182085ccb94f9b23c867acbd399e48201df425ca49703 WatchSource:0}: Error finding container dde927857a3ee098a63182085ccb94f9b23c867acbd399e48201df425ca49703: Status 404 returned error can't find the container with id dde927857a3ee098a63182085ccb94f9b23c867acbd399e48201df425ca49703 Jul 15 19:01:53.942384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.940916 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tbdkm"] Jul 15 19:01:53.952898 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:53.952797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:54.023925 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:54.023899 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b8f985_f6a5_4b0b_9588_663c9aab1550.slice/crio-481f2c7c82fe26b361eb6a50c0e67e7e965fafb3d9ccc7b9185857f3239b3a5a WatchSource:0}: Error finding container 481f2c7c82fe26b361eb6a50c0e67e7e965fafb3d9ccc7b9185857f3239b3a5a: Status 404 returned error can't find the container with id 481f2c7c82fe26b361eb6a50c0e67e7e965fafb3d9ccc7b9185857f3239b3a5a Jul 15 19:01:54.027501 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.026569 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7fmfw"] Jul 15 19:01:54.031256 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.031194 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:54.045581 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.045557 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9"] Jul 15 19:01:54.054537 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:54.054511 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cdc045a_4eb7_4e71_86d8_57bb3b7632ca.slice/crio-49d24b12bd92a62881627a336ebe321e87a3e56fece57059ca79848f3ad4f884 WatchSource:0}: Error finding container 49d24b12bd92a62881627a336ebe321e87a3e56fece57059ca79848f3ad4f884: Status 404 returned error can't find the container with id 49d24b12bd92a62881627a336ebe321e87a3e56fece57059ca79848f3ad4f884 Jul 15 19:01:54.082783 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.082759 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q"] Jul 15 19:01:54.085116 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:54.085093 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1febcb52_5110_4f8c_89b0_95338089a9bd.slice/crio-fb9cec36c969bd363b5968bd2b855e53ad6154027ee8addf943af36f930422c2 WatchSource:0}: Error finding container fb9cec36c969bd363b5968bd2b855e53ad6154027ee8addf943af36f930422c2: Status 404 returned error can't find the container with id fb9cec36c969bd363b5968bd2b855e53ad6154027ee8addf943af36f930422c2 Jul 15 19:01:54.114521 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.114471 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-94b8f45fb-n2shh"] Jul 15 19:01:54.117462 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:54.117438 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40cce77_bb87_40d5_82b0_b8c7708ec552.slice/crio-89fcb7f1ad362a35ecf60dca4117cf8afab73492a14133fc3f9c2ae1b047282b WatchSource:0}: Error finding container 89fcb7f1ad362a35ecf60dca4117cf8afab73492a14133fc3f9c2ae1b047282b: Status 404 returned error can't find the container with id 89fcb7f1ad362a35ecf60dca4117cf8afab73492a14133fc3f9c2ae1b047282b Jul 15 19:01:54.150769 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.150747 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-67swt"] Jul 15 19:01:54.154387 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:01:54.154056 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7aee3e_b9b2_46d9_ad33_15bf10f3bb34.slice/crio-f1af2a85eee86539255507aadbd1df148fb0772b71d4efc65db04768ad496df3 WatchSource:0}: Error finding container f1af2a85eee86539255507aadbd1df148fb0772b71d4efc65db04768ad496df3: Status 404 returned error can't find the container with id f1af2a85eee86539255507aadbd1df148fb0772b71d4efc65db04768ad496df3 Jul 15 19:01:54.475017 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.474974 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8shls" event={"ID":"58ff17fe-ed94-43f3-9798-816ae901a86f","Type":"ContainerStarted","Data":"fcfc1287d9360a683330624b94d371d2dc98c9e5290912d2960282454f05538c"} Jul 15 19:01:54.476306 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.476283 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" event={"ID":"c40cce77-bb87-40d5-82b0-b8c7708ec552","Type":"ContainerStarted","Data":"718f097ff0a4cf520e2965ce26e48b9c23718ca4ac00f53e10be59aa7aaf0825"} Jul 15 19:01:54.476422 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.476321 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" event={"ID":"c40cce77-bb87-40d5-82b0-b8c7708ec552","Type":"ContainerStarted","Data":"89fcb7f1ad362a35ecf60dca4117cf8afab73492a14133fc3f9c2ae1b047282b"} Jul 15 19:01:54.476422 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.476372 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:01:54.477402 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.477384 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-67swt" event={"ID":"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34","Type":"ContainerStarted","Data":"f1af2a85eee86539255507aadbd1df148fb0772b71d4efc65db04768ad496df3"} Jul 15 19:01:54.478637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.478608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fffbr" event={"ID":"cb3da942-a7bc-47c9-9bdd-578463675d5e","Type":"ContainerStarted","Data":"dde927857a3ee098a63182085ccb94f9b23c867acbd399e48201df425ca49703"} Jul 15 19:01:54.479734 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.479717 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" event={"ID":"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca","Type":"ContainerStarted","Data":"49d24b12bd92a62881627a336ebe321e87a3e56fece57059ca79848f3ad4f884"} Jul 15 19:01:54.480800 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.480770 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tbdkm" event={"ID":"00aa12c5-a721-4c14-8c1c-10c35a916c4d","Type":"ContainerStarted","Data":"c478488c9fbfcb1ab11afe23d9881806d7265318d0422c76dfb0f88b2e29a526"} Jul 15 19:01:54.481814 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.481791 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" event={"ID":"1febcb52-5110-4f8c-89b0-95338089a9bd","Type":"ContainerStarted","Data":"fb9cec36c969bd363b5968bd2b855e53ad6154027ee8addf943af36f930422c2"} Jul 15 19:01:54.482817 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.482800 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" event={"ID":"c8b8f985-f6a5-4b0b-9588-663c9aab1550","Type":"ContainerStarted","Data":"481f2c7c82fe26b361eb6a50c0e67e7e965fafb3d9ccc7b9185857f3239b3a5a"} Jul 15 19:01:54.512577 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.512525 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" podStartSLOduration=1.5125079160000001 podStartE2EDuration="1.512507916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:54.512338959 +0000 UTC m=+66.907694930" watchObservedRunningTime="2026-07-15 19:01:54.512507916 +0000 UTC m=+66.907863864" Jul 15 19:01:54.513274 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:54.513217 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8shls" podStartSLOduration=17.862716763 podStartE2EDuration="31.51320387s" totalImagesPullingTime="13.650487107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:54.493881909 +0000 UTC m=+66.889237884" watchObservedRunningTime="2026-07-15 19:01:54.51320387 +0000 UTC m=+66.908559841" Jul 15 19:01:56.449833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:56.449798 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mw954" Jul 15 19:01:57.208069 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:57.208028 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:57.306869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:57.306833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:57.365699 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:57.365640 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:57.494497 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:57.494467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:57.867353 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:57.867134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:57.932013 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:57.931934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:58.269686 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:58.269642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:58.331557 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:58.331492 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:58.678932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:58.678899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:58.779931 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:58.779875 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:58.880335 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:58.880293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:59.696946 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:59.696878 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:01:59.798377 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:01:59.798287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:00.513185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.513148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-67swt" event={"ID":"1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34","Type":"ContainerStarted","Data":"ef2fb9999c828a73255eef2418392458a304b5729a08cb82126e6ddfc93ebe39"} Jul 15 19:02:00.515303 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.515273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fffbr" event={"ID":"cb3da942-a7bc-47c9-9bdd-578463675d5e","Type":"ContainerStarted","Data":"20636ad56b15f05877a384e92ed92efba02cf7ff84aac51bc4c584a3e2bddb98"} Jul 15 19:02:00.515431 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.515324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fffbr" event={"ID":"cb3da942-a7bc-47c9-9bdd-578463675d5e","Type":"ContainerStarted","Data":"8f3d9b532248bb8fd7514059d6bd00611e23994ef14476af67cff4dcb53c4fc5"} Jul 15 19:02:00.516861 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.516822 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tbdkm" event={"ID":"00aa12c5-a721-4c14-8c1c-10c35a916c4d","Type":"ContainerStarted","Data":"af03d1fb95901e6d120c38d856074627fd30b3af8a4366a5f1abf44b8862b48d"} Jul 15 19:02:00.516982 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.516915 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:02:00.518440 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.518417 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" event={"ID":"c8b8f985-f6a5-4b0b-9588-663c9aab1550","Type":"ContainerStarted","Data":"d62d07ea4188e75358aa00817845f3f35d54de9ed993ee1788e0f73ae3354cc2"} Jul 15 19:02:00.545602 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.545556 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-67swt" podStartSLOduration=67.261245648 podStartE2EDuration="1m12.545537807s" totalImagesPullingTime="5.284292159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:00.543944309 +0000 UTC m=+72.939300280" watchObservedRunningTime="2026-07-15 19:02:00.545537807 +0000 UTC m=+72.940893779" Jul 15 19:02:00.574545 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.574505 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-7fmfw" podStartSLOduration=2.160680207 podStartE2EDuration="7.574489726s" totalImagesPullingTime="5.413809519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:00.574353207 +0000 UTC m=+72.969709190" watchObservedRunningTime="2026-07-15 19:02:00.574489726 +0000 UTC m=+72.969845696" Jul 15 19:02:00.601947 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.601887 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tbdkm" podStartSLOduration=67.11475704 podStartE2EDuration="1m12.601868572s" totalImagesPullingTime="5.487111532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:00.601511462 +0000 UTC m=+72.996867432" watchObservedRunningTime="2026-07-15 19:02:00.601868572 +0000 UTC m=+72.997224567" Jul 15 19:02:00.628303 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.628256 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fffbr" podStartSLOduration=67.081378262 podStartE2EDuration="1m12.628219216s" totalImagesPullingTime="5.546840954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:00.627993505 +0000 UTC m=+73.023349477" watchObservedRunningTime="2026-07-15 19:02:00.628219216 +0000 UTC m=+73.023575185" Jul 15 19:02:00.672344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.672308 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:00.744815 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.744771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:00.809529 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.809444 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:00.876229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.876195 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:00.977409 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:00.977377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:01.978425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:01.978394 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:02.110066 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:02.110028 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:02.203254 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:02.203213 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:03.575191 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:03.575159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:02:04.313705 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:04.313641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:04.370204 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:04.370160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:06.363137 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:06.363063 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:06.417029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:06.416993 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:06.540905 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:06.540870 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" event={"ID":"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca","Type":"ContainerStarted","Data":"219d997d625d7b52e364d6b497deb6cf82bf9fa9b017ceaf78d1180598264e1f"} Jul 15 19:02:06.541063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:06.540918 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" event={"ID":"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca","Type":"ContainerStarted","Data":"e60f6b37bf41b99c49a1e8f7224ceb4c4dfb841e8ac627031f9787e2c7097300"} Jul 15 19:02:06.541063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:06.540934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" event={"ID":"8cdc045a-4eb7-4e71-86d8-57bb3b7632ca","Type":"ContainerStarted","Data":"c15efba8c6091b050e5e44a9a5bb03a54ad8d68fb1d1427cdd13c2110ac73e06"} Jul 15 19:02:06.558588 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:06.558536 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bd56c7b-c9tm9" podStartSLOduration=1.804226654 podStartE2EDuration="13.558520699s" totalImagesPullingTime="11.754294045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:06.558278191 +0000 UTC m=+78.953634161" watchObservedRunningTime="2026-07-15 19:02:06.558520699 +0000 UTC m=+78.953876669" Jul 15 19:02:10.503392 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:10.503276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:10.613056 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:10.613027 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:10.899509 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:10.899420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:11.000877 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:11.000844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:11.355433 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:11.355398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:11.456385 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:11.456349 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:11.515939 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:11.515894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.119322 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.119281 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.243336 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.243272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.349666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.349579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.389785 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.389715 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5bfxw"] Jul 15 19:02:13.393069 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.393032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.395711 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.395687 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 19:02:13.396012 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.395890 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5shq7\"" Jul 15 19:02:13.396153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.396095 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 19:02:13.396261 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.396151 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 19:02:13.397100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.396908 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 19:02:13.397100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.396923 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 19:02:13.397300 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.397267 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 19:02:13.446327 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.446296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.510953 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.510911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-root\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.510978 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-sys\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.511011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4fz\" (UniqueName: \"kubernetes.io/projected/02f7be89-1806-4d3f-b535-81c3d89ca460-kube-api-access-jb4fz\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.511042 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-tls\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.511070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-wtmp\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.511095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511407 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.511148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511407 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.511186 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02f7be89-1806-4d3f-b535-81c3d89ca460-metrics-client-ca\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.511407 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.511217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-textfile\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.546491 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.546400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.611791 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-root\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.611791 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611797 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-sys\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611830 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4fz\" (UniqueName: \"kubernetes.io/projected/02f7be89-1806-4d3f-b535-81c3d89ca460-kube-api-access-jb4fz\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-tls\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-wtmp\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611911 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611944 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.611980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02f7be89-1806-4d3f-b535-81c3d89ca460-metrics-client-ca\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.612007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-textfile\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612424 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.612402 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-textfile\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.612958 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.612937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-root\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.613058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.612994 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-sys\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.613058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.612984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.613169 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.613103 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-wtmp\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.613633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.613610 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02f7be89-1806-4d3f-b535-81c3d89ca460-metrics-client-ca\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.615417 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.615398 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.615633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.615611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02f7be89-1806-4d3f-b535-81c3d89ca460-node-exporter-tls\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.623968 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.623921 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4fz\" (UniqueName: \"kubernetes.io/projected/02f7be89-1806-4d3f-b535-81c3d89ca460-kube-api-access-jb4fz\") pod \"node-exporter-5bfxw\" (UID: \"02f7be89-1806-4d3f-b535-81c3d89ca460\") " pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.705090 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.705004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bfxw" Jul 15 19:02:13.706212 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.706167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.737823 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:02:13.737792 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f7be89_1806_4d3f_b535_81c3d89ca460.slice/crio-aeac239d7221b8d3a76921b845efcee89e8638ee02b6942f6a7384ea076ec01f WatchSource:0}: Error finding container aeac239d7221b8d3a76921b845efcee89e8638ee02b6942f6a7384ea076ec01f: Status 404 returned error can't find the container with id aeac239d7221b8d3a76921b845efcee89e8638ee02b6942f6a7384ea076ec01f Jul 15 19:02:13.808467 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.808437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:13.938328 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:13.938291 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:14.033551 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.033518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:14.300225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.300136 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:14.394503 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.394465 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:14.466635 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.466533 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:14.570653 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.570560 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" event={"ID":"1febcb52-5110-4f8c-89b0-95338089a9bd","Type":"ContainerStarted","Data":"a6b4577077b69800193ca83775dd4047efc9066f6a730a65c5f45453b7fd4bad"} Jul 15 19:02:14.570820 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.570763 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:02:14.572578 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.572551 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bfxw" event={"ID":"02f7be89-1806-4d3f-b535-81c3d89ca460","Type":"ContainerStarted","Data":"aeac239d7221b8d3a76921b845efcee89e8638ee02b6942f6a7384ea076ec01f"} Jul 15 19:02:14.572877 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.572838 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" Jul 15 19:02:14.599432 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.599377 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57f7cc866-6s54q" podStartSLOduration=1.910939647 podStartE2EDuration="21.599355617s" totalImagesPullingTime="19.68841597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:14.59779748 +0000 UTC m=+86.993153465" watchObservedRunningTime="2026-07-15 19:02:14.599355617 +0000 UTC m=+86.994711590" Jul 15 19:02:14.814566 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:14.814538 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:15.491425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:15.491401 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-94b8f45fb-n2shh" Jul 15 19:02:15.577699 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:15.577661 2582 generic.go:356] "Generic (PLEG): container finished" podID="02f7be89-1806-4d3f-b535-81c3d89ca460" containerID="8f66c812134390a6c7a349781285926b177be805a4d31f5619e932bf19642706" exitCode=0 Jul 15 19:02:15.577840 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:15.577742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bfxw" event={"ID":"02f7be89-1806-4d3f-b535-81c3d89ca460","Type":"ContainerDied","Data":"8f66c812134390a6c7a349781285926b177be805a4d31f5619e932bf19642706"} Jul 15 19:02:15.639458 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:15.639431 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:15.739189 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:15.739155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:16.052973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.052930 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:16.148608 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.148559 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:16.257052 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.257016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:16.365174 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.365083 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:16.570870 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.570834 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:16.583497 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.583464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bfxw" event={"ID":"02f7be89-1806-4d3f-b535-81c3d89ca460","Type":"ContainerStarted","Data":"47f56c2ffc8754753826e88c5f6bac607f3fdcf28ac66975623499ed33f3d7ca"} Jul 15 19:02:16.583639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.583525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bfxw" event={"ID":"02f7be89-1806-4d3f-b535-81c3d89ca460","Type":"ContainerStarted","Data":"255979d10a5107c788fa4fcc81009df3bd97ad9485ccfcea9d3b78bf4c3e8bba"} Jul 15 19:02:16.615885 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.615785 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5bfxw" podStartSLOduration=2.47899125 podStartE2EDuration="3.615768761s" totalImagesPullingTime="1.070629064s" totalInitContainerRuntime="66.148447ms" isStatefulPod=true podCreationTimestamp="2026-07-15 19:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:16.614347375 +0000 UTC m=+89.009703349" watchObservedRunningTime="2026-07-15 19:02:16.615768761 +0000 UTC m=+89.011124730" Jul 15 19:02:16.772486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:16.772450 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:18.114220 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:18.114182 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:18.214973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:18.214924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:18.323815 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:18.323784 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:18.585369 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:18.585336 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:18.590652 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:18.590621 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" podUID="30ba2f16-961e-4e01-ba43-933a25bcae99" containerName="registry" containerID="cri-o://1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b" gracePeriod=30 Jul 15 19:02:19.356086 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.356058 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:02:19.360669 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360648 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ba2f16-961e-4e01-ba43-933a25bcae99-ca-trust-extracted\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.360783 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360684 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-bound-sa-token\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.360783 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360719 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-installation-pull-secrets\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.360783 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360757 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj6r4\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-kube-api-access-fj6r4\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.360944 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360810 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-trusted-ca\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.360944 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360843 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-image-registry-private-configuration\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.360944 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360877 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-certificates\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.360944 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.360920 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") pod \"30ba2f16-961e-4e01-ba43-933a25bcae99\" (UID: \"30ba2f16-961e-4e01-ba43-933a25bcae99\") " Jul 15 19:02:19.361363 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.361227 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:02:19.361549 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.361519 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:02:19.363458 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.363431 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:02:19.363557 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.363475 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:02:19.363557 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.363522 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:02:19.363674 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.363645 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-kube-api-access-fj6r4" (OuterVolumeSpecName: "kube-api-access-fj6r4") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "kube-api-access-fj6r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:02:19.363923 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.363904 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:02:19.369947 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.369926 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ba2f16-961e-4e01-ba43-933a25bcae99-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "30ba2f16-961e-4e01-ba43-933a25bcae99" (UID: "30ba2f16-961e-4e01-ba43-933a25bcae99"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:02:19.461558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461524 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-certificates\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.461558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461551 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-registry-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.461558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461560 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ba2f16-961e-4e01-ba43-933a25bcae99-ca-trust-extracted\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.461558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461570 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-bound-sa-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.461795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461580 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-installation-pull-secrets\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.461795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461588 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj6r4\" (UniqueName: \"kubernetes.io/projected/30ba2f16-961e-4e01-ba43-933a25bcae99-kube-api-access-fj6r4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.461795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461596 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ba2f16-961e-4e01-ba43-933a25bcae99-trusted-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.461795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.461605 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ba2f16-961e-4e01-ba43-933a25bcae99-image-registry-private-configuration\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:02:19.594072 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.594022 2582 generic.go:356] "Generic (PLEG): container finished" podID="30ba2f16-961e-4e01-ba43-933a25bcae99" containerID="1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b" exitCode=0 Jul 15 19:02:19.594072 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.594065 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" Jul 15 19:02:19.594341 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.594091 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" event={"ID":"30ba2f16-961e-4e01-ba43-933a25bcae99","Type":"ContainerDied","Data":"1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b"} Jul 15 19:02:19.594341 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.594150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bddd547bb-4sddm" event={"ID":"30ba2f16-961e-4e01-ba43-933a25bcae99","Type":"ContainerDied","Data":"a750fdd29bf14e15e4c8797fe453aad754f5406559c735c43b2796454191538f"} Jul 15 19:02:19.594341 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.594190 2582 scope.go:117] "RemoveContainer" containerID="1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b" Jul 15 19:02:19.614307 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.614286 2582 scope.go:117] "RemoveContainer" containerID="1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b" Jul 15 19:02:19.614631 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:02:19.614604 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b\": container with ID starting with 1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b not found: ID does not exist" containerID="1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b" Jul 15 19:02:19.614752 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.614636 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b"} err="failed to get container status \"1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b\": rpc error: code = NotFound desc = could not find container \"1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b\": container with ID starting with 1237e546fa472c1c5e9ecda8a00dd507830882b8f4697df8cac81df9c340f33b not found: ID does not exist" Jul 15 19:02:19.623480 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.623457 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bddd547bb-4sddm"] Jul 15 19:02:19.631421 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.631401 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bddd547bb-4sddm"] Jul 15 19:02:19.824235 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:19.824196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:20.197471 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:20.197440 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ba2f16-961e-4e01-ba43-933a25bcae99" path="/var/lib/kubelet/pods/30ba2f16-961e-4e01-ba43-933a25bcae99/volumes" Jul 15 19:02:20.219083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:20.219046 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:20.298306 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:20.298270 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:20.528172 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:20.528137 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:20.622083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:20.622050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:21.927029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:21.926985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:22.026579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:22.026545 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:26.240016 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:26.239978 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:26.340261 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:26.340210 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:28.356012 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:28.355974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:28.464109 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:28.464080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:28.556416 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:28.556379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:30.341733 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:30.341698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:30.404771 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:30.404734 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:31.537291 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:31.537255 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tbdkm" Jul 15 19:02:32.526178 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:32.526140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:32.630357 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:32.630320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:35.076206 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:35.076171 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:35.177205 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:35.177160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:35.996010 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:35.995974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:36.082989 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:36.082947 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:42.807293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:42.807235 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:42.909389 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:42.909358 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:43.009369 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:43.009340 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:59.508062 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:59.508028 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:02:59.611284 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:02:59.611222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:04.332632 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:04.332598 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:04.426546 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:04.426515 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:04.481498 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:04.481457 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:05.072296 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:05.072259 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:05.165667 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:05.165628 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:13.456645 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:13.456614 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:13.626713 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:13.626675 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:15.032921 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:15.032879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:15.121435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:15.121406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:16.816256 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:16.816212 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:20.083655 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:20.083616 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:20.180308 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:20.180277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:31.902333 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:31.902292 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:03:46.013616 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:03:46.013583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:14.562026 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:14.561985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:14.627184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:14.627147 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:15.106396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.106350 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:15.215070 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.215037 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:15.946563 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.946524 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2"] Jul 15 19:04:15.947184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.947162 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ba2f16-961e-4e01-ba43-933a25bcae99" containerName="registry" Jul 15 19:04:15.947184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.947186 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ba2f16-961e-4e01-ba43-933a25bcae99" containerName="registry" Jul 15 19:04:15.947354 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.947296 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ba2f16-961e-4e01-ba43-933a25bcae99" containerName="registry" Jul 15 19:04:15.949161 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.949142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:15.951688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.951670 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:04:15.951832 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.951814 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:04:15.952584 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.952569 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-45lbk\"" Jul 15 19:04:15.957990 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:15.957970 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2"] Jul 15 19:04:16.061478 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.061445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.061478 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.061482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.061698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.061586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82j8\" (UniqueName: \"kubernetes.io/projected/764ba17c-30cb-4117-9c08-2b1db734b8d0-kube-api-access-r82j8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.161949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.161915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.161949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.161953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.162126 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.161997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82j8\" (UniqueName: \"kubernetes.io/projected/764ba17c-30cb-4117-9c08-2b1db734b8d0-kube-api-access-r82j8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.162301 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.162283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.162301 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.162307 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.171096 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.171076 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82j8\" (UniqueName: \"kubernetes.io/projected/764ba17c-30cb-4117-9c08-2b1db734b8d0-kube-api-access-r82j8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.258360 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.258331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:16.383461 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.383428 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2"] Jul 15 19:04:16.386020 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:04:16.385992 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764ba17c_30cb_4117_9c08_2b1db734b8d0.slice/crio-284ba2ae53fc2ab8705fbac8bb02a2d8a7f237d146a2f5f84903103849ec76e4 WatchSource:0}: Error finding container 284ba2ae53fc2ab8705fbac8bb02a2d8a7f237d146a2f5f84903103849ec76e4: Status 404 returned error can't find the container with id 284ba2ae53fc2ab8705fbac8bb02a2d8a7f237d146a2f5f84903103849ec76e4 Jul 15 19:04:16.988360 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:16.988326 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" event={"ID":"764ba17c-30cb-4117-9c08-2b1db734b8d0","Type":"ContainerStarted","Data":"284ba2ae53fc2ab8705fbac8bb02a2d8a7f237d146a2f5f84903103849ec76e4"} Jul 15 19:04:26.021307 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:26.021273 2582 generic.go:356] "Generic (PLEG): container finished" podID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerID="3ff6ac7a1935fd10e82b6fa67eb2194726a9567785776c68df7a9dcdd1ffa6fe" exitCode=0 Jul 15 19:04:26.021678 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:26.021355 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" event={"ID":"764ba17c-30cb-4117-9c08-2b1db734b8d0","Type":"ContainerDied","Data":"3ff6ac7a1935fd10e82b6fa67eb2194726a9567785776c68df7a9dcdd1ffa6fe"} Jul 15 19:04:28.030231 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:28.030203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" event={"ID":"764ba17c-30cb-4117-9c08-2b1db734b8d0","Type":"ContainerStarted","Data":"e789d4b6b2a94a20dc0249e73590daef7ca80af4eade0ead37651971eb77f72c"} Jul 15 19:04:29.035135 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:29.035099 2582 generic.go:356] "Generic (PLEG): container finished" podID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerID="e789d4b6b2a94a20dc0249e73590daef7ca80af4eade0ead37651971eb77f72c" exitCode=0 Jul 15 19:04:29.035549 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:29.035148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" event={"ID":"764ba17c-30cb-4117-9c08-2b1db734b8d0","Type":"ContainerDied","Data":"e789d4b6b2a94a20dc0249e73590daef7ca80af4eade0ead37651971eb77f72c"} Jul 15 19:04:36.062710 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:36.062618 2582 generic.go:356] "Generic (PLEG): container finished" podID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerID="b861e71c3fa4af1b97af89b430c376ddf2bedae469434658e19ca56cc7520b04" exitCode=0 Jul 15 19:04:36.062710 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:36.062694 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" event={"ID":"764ba17c-30cb-4117-9c08-2b1db734b8d0","Type":"ContainerDied","Data":"b861e71c3fa4af1b97af89b430c376ddf2bedae469434658e19ca56cc7520b04"} Jul 15 19:04:37.195330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.195304 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:37.223715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.223690 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-util\") pod \"764ba17c-30cb-4117-9c08-2b1db734b8d0\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " Jul 15 19:04:37.223852 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.223749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r82j8\" (UniqueName: \"kubernetes.io/projected/764ba17c-30cb-4117-9c08-2b1db734b8d0-kube-api-access-r82j8\") pod \"764ba17c-30cb-4117-9c08-2b1db734b8d0\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " Jul 15 19:04:37.223852 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.223777 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-bundle\") pod \"764ba17c-30cb-4117-9c08-2b1db734b8d0\" (UID: \"764ba17c-30cb-4117-9c08-2b1db734b8d0\") " Jul 15 19:04:37.224556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.224526 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-bundle" (OuterVolumeSpecName: "bundle") pod "764ba17c-30cb-4117-9c08-2b1db734b8d0" (UID: "764ba17c-30cb-4117-9c08-2b1db734b8d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:04:37.226131 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.226104 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764ba17c-30cb-4117-9c08-2b1db734b8d0-kube-api-access-r82j8" (OuterVolumeSpecName: "kube-api-access-r82j8") pod "764ba17c-30cb-4117-9c08-2b1db734b8d0" (UID: "764ba17c-30cb-4117-9c08-2b1db734b8d0"). InnerVolumeSpecName "kube-api-access-r82j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:04:37.228909 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.228887 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-util" (OuterVolumeSpecName: "util") pod "764ba17c-30cb-4117-9c08-2b1db734b8d0" (UID: "764ba17c-30cb-4117-9c08-2b1db734b8d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:04:37.325314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.325205 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:04:37.325314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.325252 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764ba17c-30cb-4117-9c08-2b1db734b8d0-util\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:04:37.325314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:37.325266 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r82j8\" (UniqueName: \"kubernetes.io/projected/764ba17c-30cb-4117-9c08-2b1db734b8d0-kube-api-access-r82j8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:04:38.071024 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:38.070987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" event={"ID":"764ba17c-30cb-4117-9c08-2b1db734b8d0","Type":"ContainerDied","Data":"284ba2ae53fc2ab8705fbac8bb02a2d8a7f237d146a2f5f84903103849ec76e4"} Jul 15 19:04:38.071184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:38.071031 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dlkc2" Jul 15 19:04:38.071184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:38.071047 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284ba2ae53fc2ab8705fbac8bb02a2d8a7f237d146a2f5f84903103849ec76e4" Jul 15 19:04:42.059273 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:42.059221 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:42.907938 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:42.907901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:45.928652 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:45.928614 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:46.027177 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:46.027142 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:47.780853 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:47.780805 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:47.882126 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:47.882090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:48.740348 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:48.740309 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:48.869356 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:48.869308 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:49.820825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:49.820775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:50.248724 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:50.248687 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:52.862804 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:52.862766 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:52.965803 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:52.965776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:53.717173 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:53.717121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:04:58.399011 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:04:58.398951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:15.096855 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:15.096754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:15.194908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:15.194873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:16.678865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.678829 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g"] Jul 15 19:05:16.679269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.679063 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerName="util" Jul 15 19:05:16.679269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.679074 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerName="util" Jul 15 19:05:16.679269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.679081 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerName="pull" Jul 15 19:05:16.679269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.679086 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerName="pull" Jul 15 19:05:16.679269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.679094 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerName="extract" Jul 15 19:05:16.679269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.679099 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerName="extract" Jul 15 19:05:16.679269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.679142 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="764ba17c-30cb-4117-9c08-2b1db734b8d0" containerName="extract" Jul 15 19:05:16.681087 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.681069 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:16.684167 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.684147 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vkzzq\"" Jul 15 19:05:16.684882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.684866 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:05:16.684971 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.684898 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 19:05:16.693271 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.693249 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g"] Jul 15 19:05:16.802760 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.802723 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw78h\" (UniqueName: \"kubernetes.io/projected/c836347c-92d2-415d-bfd5-26087755e779-kube-api-access-tw78h\") pod \"openshift-lws-operator-fd8ccff4c-75z8g\" (UID: \"c836347c-92d2-415d-bfd5-26087755e779\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:16.802918 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.802822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c836347c-92d2-415d-bfd5-26087755e779-tmp\") pod \"openshift-lws-operator-fd8ccff4c-75z8g\" (UID: \"c836347c-92d2-415d-bfd5-26087755e779\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:16.903998 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.903962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw78h\" (UniqueName: \"kubernetes.io/projected/c836347c-92d2-415d-bfd5-26087755e779-kube-api-access-tw78h\") pod \"openshift-lws-operator-fd8ccff4c-75z8g\" (UID: \"c836347c-92d2-415d-bfd5-26087755e779\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:16.904180 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.904036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c836347c-92d2-415d-bfd5-26087755e779-tmp\") pod \"openshift-lws-operator-fd8ccff4c-75z8g\" (UID: \"c836347c-92d2-415d-bfd5-26087755e779\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:16.904425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.904405 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c836347c-92d2-415d-bfd5-26087755e779-tmp\") pod \"openshift-lws-operator-fd8ccff4c-75z8g\" (UID: \"c836347c-92d2-415d-bfd5-26087755e779\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:16.912013 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.911994 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw78h\" (UniqueName: \"kubernetes.io/projected/c836347c-92d2-415d-bfd5-26087755e779-kube-api-access-tw78h\") pod \"openshift-lws-operator-fd8ccff4c-75z8g\" (UID: \"c836347c-92d2-415d-bfd5-26087755e779\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:16.990101 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:16.990076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" Jul 15 19:05:17.121581 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:17.121550 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g"] Jul 15 19:05:17.125405 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:05:17.125378 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc836347c_92d2_415d_bfd5_26087755e779.slice/crio-81806ed2879283463b35fa3a07ebe2ce99ab78e71c26c065a9e062f1f39b3f92 WatchSource:0}: Error finding container 81806ed2879283463b35fa3a07ebe2ce99ab78e71c26c065a9e062f1f39b3f92: Status 404 returned error can't find the container with id 81806ed2879283463b35fa3a07ebe2ce99ab78e71c26c065a9e062f1f39b3f92 Jul 15 19:05:17.203864 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:17.203834 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" event={"ID":"c836347c-92d2-415d-bfd5-26087755e779","Type":"ContainerStarted","Data":"81806ed2879283463b35fa3a07ebe2ce99ab78e71c26c065a9e062f1f39b3f92"} Jul 15 19:05:17.555280 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:17.555224 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:17.663534 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:17.663498 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:20.216215 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:20.216177 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" event={"ID":"c836347c-92d2-415d-bfd5-26087755e779","Type":"ContainerStarted","Data":"cb7e008b17dff66af99b858c9cc64a9cce5174100e473fbcb6019aca14564747"} Jul 15 19:05:20.233661 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:20.233611 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-75z8g" podStartSLOduration=2.055479097 podStartE2EDuration="4.233591115s" totalImagesPullingTime="2.178112018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:05:20.233203809 +0000 UTC m=+272.628559782" watchObservedRunningTime="2026-07-15 19:05:20.233591115 +0000 UTC m=+272.628947086" Jul 15 19:05:20.958694 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:20.958654 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:21.062289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:21.062260 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:22.648302 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.648264 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp"] Jul 15 19:05:22.650374 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.650358 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.652823 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.652792 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-45lbk\"" Jul 15 19:05:22.652927 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.652810 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:05:22.653615 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.653601 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:05:22.659599 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.659580 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp"] Jul 15 19:05:22.753576 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.753537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlg46\" (UniqueName: \"kubernetes.io/projected/17af6aea-3ee4-479f-b161-f62d26d6839b-kube-api-access-dlg46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.753715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.753584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.753715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.753610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.854376 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.854343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlg46\" (UniqueName: \"kubernetes.io/projected/17af6aea-3ee4-479f-b161-f62d26d6839b-kube-api-access-dlg46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.854542 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.854394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.854542 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.854524 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.854791 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.854772 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.854836 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.854821 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.863051 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.863025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlg46\" (UniqueName: \"kubernetes.io/projected/17af6aea-3ee4-479f-b161-f62d26d6839b-kube-api-access-dlg46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:22.918730 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.918651 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:22.959817 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:22.959780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:23.085114 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:23.085090 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp"] Jul 15 19:05:23.087289 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:05:23.087262 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17af6aea_3ee4_479f_b161_f62d26d6839b.slice/crio-fc25d83ae928770871fce5d7e0a0ab04001be1c4828707da9e326dde50081712 WatchSource:0}: Error finding container fc25d83ae928770871fce5d7e0a0ab04001be1c4828707da9e326dde50081712: Status 404 returned error can't find the container with id fc25d83ae928770871fce5d7e0a0ab04001be1c4828707da9e326dde50081712 Jul 15 19:05:23.228756 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:23.228721 2582 generic.go:356] "Generic (PLEG): container finished" podID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerID="7299602fcd2acbcae83c22231df6ebc7b57a7cbbd1cfea4a6419aa3dcca1811f" exitCode=0 Jul 15 19:05:23.228942 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:23.228780 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" event={"ID":"17af6aea-3ee4-479f-b161-f62d26d6839b","Type":"ContainerDied","Data":"7299602fcd2acbcae83c22231df6ebc7b57a7cbbd1cfea4a6419aa3dcca1811f"} Jul 15 19:05:23.228942 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:23.228844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" event={"ID":"17af6aea-3ee4-479f-b161-f62d26d6839b","Type":"ContainerStarted","Data":"fc25d83ae928770871fce5d7e0a0ab04001be1c4828707da9e326dde50081712"} Jul 15 19:05:25.238384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:25.238354 2582 generic.go:356] "Generic (PLEG): container finished" podID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerID="a394976f72d7c8d2cc95b37dcc1aafde7473b31ce5bd0945d9fda131d521c0a4" exitCode=0 Jul 15 19:05:25.238764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:25.238439 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" event={"ID":"17af6aea-3ee4-479f-b161-f62d26d6839b","Type":"ContainerDied","Data":"a394976f72d7c8d2cc95b37dcc1aafde7473b31ce5bd0945d9fda131d521c0a4"} Jul 15 19:05:26.244135 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:26.244095 2582 generic.go:356] "Generic (PLEG): container finished" podID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerID="534c81394cef2d5e4cf6173019a55f6673532c86de345de1d1d55e5a9f2290f1" exitCode=0 Jul 15 19:05:26.244598 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:26.244196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" event={"ID":"17af6aea-3ee4-479f-b161-f62d26d6839b","Type":"ContainerDied","Data":"534c81394cef2d5e4cf6173019a55f6673532c86de345de1d1d55e5a9f2290f1"} Jul 15 19:05:27.370214 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.370186 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:27.488278 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.488229 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-util\") pod \"17af6aea-3ee4-479f-b161-f62d26d6839b\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " Jul 15 19:05:27.488452 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.488294 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlg46\" (UniqueName: \"kubernetes.io/projected/17af6aea-3ee4-479f-b161-f62d26d6839b-kube-api-access-dlg46\") pod \"17af6aea-3ee4-479f-b161-f62d26d6839b\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " Jul 15 19:05:27.488452 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.488333 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-bundle\") pod \"17af6aea-3ee4-479f-b161-f62d26d6839b\" (UID: \"17af6aea-3ee4-479f-b161-f62d26d6839b\") " Jul 15 19:05:27.489038 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.489007 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-bundle" (OuterVolumeSpecName: "bundle") pod "17af6aea-3ee4-479f-b161-f62d26d6839b" (UID: "17af6aea-3ee4-479f-b161-f62d26d6839b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:05:27.490398 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.490374 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17af6aea-3ee4-479f-b161-f62d26d6839b-kube-api-access-dlg46" (OuterVolumeSpecName: "kube-api-access-dlg46") pod "17af6aea-3ee4-479f-b161-f62d26d6839b" (UID: "17af6aea-3ee4-479f-b161-f62d26d6839b"). InnerVolumeSpecName "kube-api-access-dlg46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:05:27.494163 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.494121 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-util" (OuterVolumeSpecName: "util") pod "17af6aea-3ee4-479f-b161-f62d26d6839b" (UID: "17af6aea-3ee4-479f-b161-f62d26d6839b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:05:27.588992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.588912 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlg46\" (UniqueName: \"kubernetes.io/projected/17af6aea-3ee4-479f-b161-f62d26d6839b-kube-api-access-dlg46\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:05:27.588992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.588949 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:05:27.588992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:27.588960 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17af6aea-3ee4-479f-b161-f62d26d6839b-util\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:05:28.252349 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:28.252315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" event={"ID":"17af6aea-3ee4-479f-b161-f62d26d6839b","Type":"ContainerDied","Data":"fc25d83ae928770871fce5d7e0a0ab04001be1c4828707da9e326dde50081712"} Jul 15 19:05:28.252495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:28.252351 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5pvvp" Jul 15 19:05:28.252495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:28.252366 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc25d83ae928770871fce5d7e0a0ab04001be1c4828707da9e326dde50081712" Jul 15 19:05:35.171816 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:35.171775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:35.394268 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:35.394220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:37.989402 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:37.989363 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:38.960988 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.960953 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt"] Jul 15 19:05:38.961214 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.961203 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerName="pull" Jul 15 19:05:38.961293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.961216 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerName="pull" Jul 15 19:05:38.961293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.961234 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerName="util" Jul 15 19:05:38.961293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.961254 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerName="util" Jul 15 19:05:38.961293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.961267 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerName="extract" Jul 15 19:05:38.961293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.961275 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerName="extract" Jul 15 19:05:38.961444 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.961318 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="17af6aea-3ee4-479f-b161-f62d26d6839b" containerName="extract" Jul 15 19:05:38.963492 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.963476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:38.969184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.969157 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 19:05:38.969310 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.969163 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 19:05:38.969372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.969355 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 19:05:38.969783 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.969703 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rf8qx\"" Jul 15 19:05:38.970971 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.970953 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 19:05:38.997106 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:38.997080 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt"] Jul 15 19:05:39.072043 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.072013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8779e7fe-5d58-4525-a675-f3a82e5996f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.072195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.072059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8779e7fe-5d58-4525-a675-f3a82e5996f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.072195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.072143 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8779e7fe-5d58-4525-a675-f3a82e5996f5-manifests\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.072195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.072176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ncm\" (UniqueName: \"kubernetes.io/projected/8779e7fe-5d58-4525-a675-f3a82e5996f5-kube-api-access-c4ncm\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.072335 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.072201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8779e7fe-5d58-4525-a675-f3a82e5996f5-tmp\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.173469 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.173433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8779e7fe-5d58-4525-a675-f3a82e5996f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.173629 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.173476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8779e7fe-5d58-4525-a675-f3a82e5996f5-manifests\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.173629 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.173499 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ncm\" (UniqueName: \"kubernetes.io/projected/8779e7fe-5d58-4525-a675-f3a82e5996f5-kube-api-access-c4ncm\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.173629 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.173518 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8779e7fe-5d58-4525-a675-f3a82e5996f5-tmp\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.173629 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.173554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8779e7fe-5d58-4525-a675-f3a82e5996f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.173866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.173839 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8779e7fe-5d58-4525-a675-f3a82e5996f5-manifests\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.175826 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.175804 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8779e7fe-5d58-4525-a675-f3a82e5996f5-tmp\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.176051 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.176034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8779e7fe-5d58-4525-a675-f3a82e5996f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.176092 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.176036 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8779e7fe-5d58-4525-a675-f3a82e5996f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.181638 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.181614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ncm\" (UniqueName: \"kubernetes.io/projected/8779e7fe-5d58-4525-a675-f3a82e5996f5-kube-api-access-c4ncm\") pod \"opendatahub-operator-controller-manager-55c89bff45-6njpt\" (UID: \"8779e7fe-5d58-4525-a675-f3a82e5996f5\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.273709 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.273675 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:39.406981 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.406937 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt"] Jul 15 19:05:39.410784 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:05:39.410759 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8779e7fe_5d58_4525_a675_f3a82e5996f5.slice/crio-aa62e2874fe625e73e52bfefd912abb7b87b34efe24f47a3f74625637f617c03 WatchSource:0}: Error finding container aa62e2874fe625e73e52bfefd912abb7b87b34efe24f47a3f74625637f617c03: Status 404 returned error can't find the container with id aa62e2874fe625e73e52bfefd912abb7b87b34efe24f47a3f74625637f617c03 Jul 15 19:05:39.491453 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.491424 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:39.594931 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.594847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:39.699063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:39.699030 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:40.298788 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:40.298751 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" event={"ID":"8779e7fe-5d58-4525-a675-f3a82e5996f5","Type":"ContainerStarted","Data":"aa62e2874fe625e73e52bfefd912abb7b87b34efe24f47a3f74625637f617c03"} Jul 15 19:05:41.139331 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:41.139144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:41.246102 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:41.246055 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:43.311315 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:43.311277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" event={"ID":"8779e7fe-5d58-4525-a675-f3a82e5996f5","Type":"ContainerStarted","Data":"302e735547ed3f008d6b7bc91f3572aca50085f8e39544321d818d511af61628"} Jul 15 19:05:44.316376 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:44.316343 2582 generic.go:356] "Generic (PLEG): container finished" podID="8779e7fe-5d58-4525-a675-f3a82e5996f5" containerID="302e735547ed3f008d6b7bc91f3572aca50085f8e39544321d818d511af61628" exitCode=0 Jul 15 19:05:44.316836 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:44.316383 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" event={"ID":"8779e7fe-5d58-4525-a675-f3a82e5996f5","Type":"ContainerDied","Data":"302e735547ed3f008d6b7bc91f3572aca50085f8e39544321d818d511af61628"} Jul 15 19:05:45.321943 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:45.321844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" event={"ID":"8779e7fe-5d58-4525-a675-f3a82e5996f5","Type":"ContainerStarted","Data":"fbe59e4086ca5f6edb1d5c61dcb18e2e8835faea348610a55e3584464a5dbed4"} Jul 15 19:05:45.322408 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:45.321978 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:45.351593 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:45.351536 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" podStartSLOduration=2.218671118 podStartE2EDuration="7.351519775s" totalImagesPullingTime="4.07180633s" totalInitContainerRuntime="1.061042327s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:05:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:05:45.349530413 +0000 UTC m=+297.744886382" watchObservedRunningTime="2026-07-15 19:05:45.351519775 +0000 UTC m=+297.746875744" Jul 15 19:05:48.035119 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:48.035087 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:05:48.038329 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:48.038298 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:05:48.042013 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:48.041993 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 19:05:54.036984 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:54.036946 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:54.100977 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:54.100951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:54.462204 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:54.462110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:54.523226 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:54.523187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:56.328321 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:56.328291 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-6njpt" Jul 15 19:05:58.264456 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.264419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:58.418569 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.418529 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:58.896028 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.895991 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:58.910937 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.910903 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn"] Jul 15 19:05:58.913438 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.913420 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:58.918678 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.918655 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 19:05:58.918838 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.918822 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 19:05:58.919074 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.919058 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7hxkf\"" Jul 15 19:05:58.921404 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.921388 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 19:05:58.921881 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.921867 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 19:05:58.941833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:58.941810 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn"] Jul 15 19:05:59.013613 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.013582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82x24\" (UniqueName: \"kubernetes.io/projected/36114d69-e0aa-4dc6-b314-0ed3c1214b69-kube-api-access-82x24\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.013761 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.013619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36114d69-e0aa-4dc6-b314-0ed3c1214b69-tls-certs\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.013761 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.013656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36114d69-e0aa-4dc6-b314-0ed3c1214b69-tmp\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.107740 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.107712 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7gvhd"] Jul 15 19:05:59.109858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.109842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:05:59.114786 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.114755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x24\" (UniqueName: \"kubernetes.io/projected/36114d69-e0aa-4dc6-b314-0ed3c1214b69-kube-api-access-82x24\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.114909 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.114803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36114d69-e0aa-4dc6-b314-0ed3c1214b69-tls-certs\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.114909 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.114863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36114d69-e0aa-4dc6-b314-0ed3c1214b69-tmp\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.115314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.115293 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bdgr9\"" Jul 15 19:05:59.117194 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.117173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36114d69-e0aa-4dc6-b314-0ed3c1214b69-tmp\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.117616 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.117596 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36114d69-e0aa-4dc6-b314-0ed3c1214b69-tls-certs\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.120335 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.120317 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 19:05:59.137039 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.137015 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7gvhd"] Jul 15 19:05:59.139384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.139363 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x24\" (UniqueName: \"kubernetes.io/projected/36114d69-e0aa-4dc6-b314-0ed3c1214b69-kube-api-access-82x24\") pod \"kube-auth-proxy-54f9c67c7d-x6ccn\" (UID: \"36114d69-e0aa-4dc6-b314-0ed3c1214b69\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.146506 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.146439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:59.215828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.215801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eefbafad-7ce8-4f41-bcb9-2223375f54f7-cert\") pod \"odh-model-controller-7998c67dc4-7gvhd\" (UID: \"eefbafad-7ce8-4f41-bcb9-2223375f54f7\") " pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:05:59.215952 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.215838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gnn2\" (UniqueName: \"kubernetes.io/projected/eefbafad-7ce8-4f41-bcb9-2223375f54f7-kube-api-access-2gnn2\") pod \"odh-model-controller-7998c67dc4-7gvhd\" (UID: \"eefbafad-7ce8-4f41-bcb9-2223375f54f7\") " pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:05:59.222787 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.222766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" Jul 15 19:05:59.248075 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.248041 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:59.316668 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.316636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eefbafad-7ce8-4f41-bcb9-2223375f54f7-cert\") pod \"odh-model-controller-7998c67dc4-7gvhd\" (UID: \"eefbafad-7ce8-4f41-bcb9-2223375f54f7\") " pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:05:59.316972 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.316695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnn2\" (UniqueName: \"kubernetes.io/projected/eefbafad-7ce8-4f41-bcb9-2223375f54f7-kube-api-access-2gnn2\") pod \"odh-model-controller-7998c67dc4-7gvhd\" (UID: \"eefbafad-7ce8-4f41-bcb9-2223375f54f7\") " pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:05:59.316972 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:05:59.316772 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 19:05:59.316972 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:05:59.316856 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eefbafad-7ce8-4f41-bcb9-2223375f54f7-cert podName:eefbafad-7ce8-4f41-bcb9-2223375f54f7 nodeName:}" failed. No retries permitted until 2026-07-15 19:05:59.816835045 +0000 UTC m=+312.212191007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/eefbafad-7ce8-4f41-bcb9-2223375f54f7-cert") pod "odh-model-controller-7998c67dc4-7gvhd" (UID: "eefbafad-7ce8-4f41-bcb9-2223375f54f7") : secret "odh-model-controller-webhook-cert" not found Jul 15 19:05:59.333495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.333463 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnn2\" (UniqueName: \"kubernetes.io/projected/eefbafad-7ce8-4f41-bcb9-2223375f54f7-kube-api-access-2gnn2\") pod \"odh-model-controller-7998c67dc4-7gvhd\" (UID: \"eefbafad-7ce8-4f41-bcb9-2223375f54f7\") " pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:05:59.376874 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.376844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:59.379820 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.379796 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn"] Jul 15 19:05:59.383675 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:05:59.383649 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36114d69_e0aa_4dc6_b314_0ed3c1214b69.slice/crio-d09b99c8af2982e126619a36fc32f759be66421a43a5630024c2b8c3905e2f3a WatchSource:0}: Error finding container d09b99c8af2982e126619a36fc32f759be66421a43a5630024c2b8c3905e2f3a: Status 404 returned error can't find the container with id d09b99c8af2982e126619a36fc32f759be66421a43a5630024c2b8c3905e2f3a Jul 15 19:05:59.385393 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.385379 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:05:59.533481 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.533449 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:05:59.821064 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.820983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eefbafad-7ce8-4f41-bcb9-2223375f54f7-cert\") pod \"odh-model-controller-7998c67dc4-7gvhd\" (UID: \"eefbafad-7ce8-4f41-bcb9-2223375f54f7\") " pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:05:59.823236 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:05:59.823217 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eefbafad-7ce8-4f41-bcb9-2223375f54f7-cert\") pod \"odh-model-controller-7998c67dc4-7gvhd\" (UID: \"eefbafad-7ce8-4f41-bcb9-2223375f54f7\") " pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:06:00.019505 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.019472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:06:00.085673 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.085598 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:00.156252 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.156210 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-slhht"] Jul 15 19:06:00.159100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.159081 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.162687 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.162653 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 19:06:00.162793 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.162693 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-27dx2\"" Jul 15 19:06:00.164720 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.164655 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7gvhd"] Jul 15 19:06:00.168380 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:06:00.168356 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefbafad_7ce8_4f41_bcb9_2223375f54f7.slice/crio-39a4b85aaf5726d22688ad6c2327905e9d34b5d2e8f6bc953020807ea8956ef5 WatchSource:0}: Error finding container 39a4b85aaf5726d22688ad6c2327905e9d34b5d2e8f6bc953020807ea8956ef5: Status 404 returned error can't find the container with id 39a4b85aaf5726d22688ad6c2327905e9d34b5d2e8f6bc953020807ea8956ef5 Jul 15 19:06:00.184314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.184286 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-slhht"] Jul 15 19:06:00.219024 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.219000 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:00.224024 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.224005 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1673566b-590c-429b-bb0f-d83027cd493f-tls-certs\") pod \"model-serving-api-54d89bd5fc-slhht\" (UID: \"1673566b-590c-429b-bb0f-d83027cd493f\") " pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.224085 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.224040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqcb\" (UniqueName: \"kubernetes.io/projected/1673566b-590c-429b-bb0f-d83027cd493f-kube-api-access-kpqcb\") pod \"model-serving-api-54d89bd5fc-slhht\" (UID: \"1673566b-590c-429b-bb0f-d83027cd493f\") " pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.324830 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.324796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1673566b-590c-429b-bb0f-d83027cd493f-tls-certs\") pod \"model-serving-api-54d89bd5fc-slhht\" (UID: \"1673566b-590c-429b-bb0f-d83027cd493f\") " pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.325312 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.324843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqcb\" (UniqueName: \"kubernetes.io/projected/1673566b-590c-429b-bb0f-d83027cd493f-kube-api-access-kpqcb\") pod \"model-serving-api-54d89bd5fc-slhht\" (UID: \"1673566b-590c-429b-bb0f-d83027cd493f\") " pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.327665 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.327642 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1673566b-590c-429b-bb0f-d83027cd493f-tls-certs\") pod \"model-serving-api-54d89bd5fc-slhht\" (UID: \"1673566b-590c-429b-bb0f-d83027cd493f\") " pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.334571 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.334545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqcb\" (UniqueName: \"kubernetes.io/projected/1673566b-590c-429b-bb0f-d83027cd493f-kube-api-access-kpqcb\") pod \"model-serving-api-54d89bd5fc-slhht\" (UID: \"1673566b-590c-429b-bb0f-d83027cd493f\") " pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.378457 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.378370 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" event={"ID":"36114d69-e0aa-4dc6-b314-0ed3c1214b69","Type":"ContainerStarted","Data":"d09b99c8af2982e126619a36fc32f759be66421a43a5630024c2b8c3905e2f3a"} Jul 15 19:06:00.379435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.379414 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" event={"ID":"eefbafad-7ce8-4f41-bcb9-2223375f54f7","Type":"ContainerStarted","Data":"39a4b85aaf5726d22688ad6c2327905e9d34b5d2e8f6bc953020807ea8956ef5"} Jul 15 19:06:00.478050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.478014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:00.610941 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.610921 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-slhht"] Jul 15 19:06:00.613127 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:06:00.613101 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1673566b_590c_429b_bb0f_d83027cd493f.slice/crio-cb4d1287fbff39268322b70e6267bdaa6f336d86bb08edf7cde0c5d1a58a5f38 WatchSource:0}: Error finding container cb4d1287fbff39268322b70e6267bdaa6f336d86bb08edf7cde0c5d1a58a5f38: Status 404 returned error can't find the container with id cb4d1287fbff39268322b70e6267bdaa6f336d86bb08edf7cde0c5d1a58a5f38 Jul 15 19:06:00.650203 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.650138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:00.660447 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.660424 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rptvn"] Jul 15 19:06:00.664235 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.664217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.666981 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.666955 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 19:06:00.667075 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.666988 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 19:06:00.667429 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.667413 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sgqpr\"" Jul 15 19:06:00.667552 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.667527 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 19:06:00.667956 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.667937 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 19:06:00.683189 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.683168 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rptvn"] Jul 15 19:06:00.728917 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.728879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.729058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.728927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-token\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.729058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.728951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-cacerts\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.729058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.729026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.729192 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.729069 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskkm\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-kube-api-access-sskkm\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.729192 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.729101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ee25d-9f31-41ab-a573-5de73fbb03d9-local-certs\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.729192 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.729133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.751353 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.751313 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:00.830486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.830455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.830654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.830493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-token\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.830654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.830510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-cacerts\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.830654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.830541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.830654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.830562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskkm\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-kube-api-access-sskkm\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.830654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.830582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ee25d-9f31-41ab-a573-5de73fbb03d9-local-certs\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.830654 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.830604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.831355 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.831307 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.832897 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.832872 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ee25d-9f31-41ab-a573-5de73fbb03d9-local-certs\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.832990 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.832943 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-cacerts\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.833038 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.832990 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.833076 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.833034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.838809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.838787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-token\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.840011 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.839988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskkm\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-kube-api-access-sskkm\") pod \"istiod-openshift-gateway-7574459656-rptvn\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:00.976493 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:00.976464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:01.111979 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:01.111947 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rptvn"] Jul 15 19:06:01.115271 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:06:01.115222 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54ee25d_9f31_41ab_a573_5de73fbb03d9.slice/crio-a3de29d58c215b86e0b85375b66a1d1de629374b7c35b1fb3f91aecce5c4bd58 WatchSource:0}: Error finding container a3de29d58c215b86e0b85375b66a1d1de629374b7c35b1fb3f91aecce5c4bd58: Status 404 returned error can't find the container with id a3de29d58c215b86e0b85375b66a1d1de629374b7c35b1fb3f91aecce5c4bd58 Jul 15 19:06:01.386444 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:01.386348 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" event={"ID":"e54ee25d-9f31-41ab-a573-5de73fbb03d9","Type":"ContainerStarted","Data":"a3de29d58c215b86e0b85375b66a1d1de629374b7c35b1fb3f91aecce5c4bd58"} Jul 15 19:06:01.387699 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:01.387629 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-slhht" event={"ID":"1673566b-590c-429b-bb0f-d83027cd493f","Type":"ContainerStarted","Data":"cb4d1287fbff39268322b70e6267bdaa6f336d86bb08edf7cde0c5d1a58a5f38"} Jul 15 19:06:02.013906 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:02.013863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:05.438713 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:05.438674 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:06:05.438981 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:05.438740 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:06:06.255811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.255773 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:06.349176 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.349138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:06.416262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.412132 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-slhht" event={"ID":"1673566b-590c-429b-bb0f-d83027cd493f","Type":"ContainerStarted","Data":"2538979e6be5825ce098a70a34f7a950991fe69ff11b2ec25721998c7c1f7183"} Jul 15 19:06:06.416262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.412747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:06.416262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.414336 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" event={"ID":"eefbafad-7ce8-4f41-bcb9-2223375f54f7","Type":"ContainerStarted","Data":"c5d9754eab3613c4121e59dce8d31639e9f53f6978ed57f44c32a431e9f937f0"} Jul 15 19:06:06.416262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.414919 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:06:06.416262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.416229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" event={"ID":"36114d69-e0aa-4dc6-b314-0ed3c1214b69","Type":"ContainerStarted","Data":"5f771fc9cdc521da5da70ca0e8b1c96cbc7ef2515390d25438759adce28ecb61"} Jul 15 19:06:06.418411 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.418381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" event={"ID":"e54ee25d-9f31-41ab-a573-5de73fbb03d9","Type":"ContainerStarted","Data":"ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7"} Jul 15 19:06:06.418591 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.418570 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:06.420106 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.420082 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-rptvn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:06:06.420181 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.420127 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" podUID="e54ee25d-9f31-41ab-a573-5de73fbb03d9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:06:06.446706 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.446664 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-slhht" podStartSLOduration=1.630849967 podStartE2EDuration="6.446650686s" totalImagesPullingTime="4.815800719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:06.445839977 +0000 UTC m=+318.841195971" watchObservedRunningTime="2026-07-15 19:06:06.446650686 +0000 UTC m=+318.842006655" Jul 15 19:06:06.479348 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.479301 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" podStartSLOduration=2.1581920549999998 podStartE2EDuration="6.479286633s" totalImagesPullingTime="4.321094578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:06.477685976 +0000 UTC m=+318.873041958" watchObservedRunningTime="2026-07-15 19:06:06.479286633 +0000 UTC m=+318.874642605" Jul 15 19:06:06.533850 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.533743 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" podStartSLOduration=2.27121849 podStartE2EDuration="7.533721327s" totalImagesPullingTime="5.262502837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:06.533087798 +0000 UTC m=+318.928443769" watchObservedRunningTime="2026-07-15 19:06:06.533721327 +0000 UTC m=+318.929077298" Jul 15 19:06:06.534406 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.534363 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-x6ccn" podStartSLOduration=2.489930846 podStartE2EDuration="8.534351365s" totalImagesPullingTime="6.044420519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:06.515376714 +0000 UTC m=+318.910732706" watchObservedRunningTime="2026-07-15 19:06:06.534351365 +0000 UTC m=+318.929707333" Jul 15 19:06:06.816306 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.816222 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl"] Jul 15 19:06:06.819566 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.819546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.823162 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.823139 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9f5t2\"" Jul 15 19:06:06.842175 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.842151 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl"] Jul 15 19:06:06.882755 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.882858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882766 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjg8\" (UniqueName: \"kubernetes.io/projected/73198a57-057f-4510-8d27-8c9349325bc9-kube-api-access-lnjg8\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.882858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.882858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882816 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73198a57-057f-4510-8d27-8c9349325bc9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.882967 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73198a57-057f-4510-8d27-8c9349325bc9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.882967 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.883023 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73198a57-057f-4510-8d27-8c9349325bc9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.883023 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.882999 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.883086 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.883026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.910947 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.910906 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:06.984333 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984494 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjg8\" (UniqueName: \"kubernetes.io/projected/73198a57-057f-4510-8d27-8c9349325bc9-kube-api-access-lnjg8\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984494 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984494 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73198a57-057f-4510-8d27-8c9349325bc9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984494 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73198a57-057f-4510-8d27-8c9349325bc9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984494 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73198a57-057f-4510-8d27-8c9349325bc9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.984801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984730 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.985100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.984887 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.985382 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.985358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73198a57-057f-4510-8d27-8c9349325bc9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.985488 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.985396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.987216 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.987193 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73198a57-057f-4510-8d27-8c9349325bc9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.987418 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.987398 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73198a57-057f-4510-8d27-8c9349325bc9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.999132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.999111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjg8\" (UniqueName: \"kubernetes.io/projected/73198a57-057f-4510-8d27-8c9349325bc9-kube-api-access-lnjg8\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:06.999383 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:06.999364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73198a57-057f-4510-8d27-8c9349325bc9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j9hcl\" (UID: \"73198a57-057f-4510-8d27-8c9349325bc9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:07.079587 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:07.079504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:07.130125 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:07.130091 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:07.179036 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:07.178966 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:07.264702 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:07.264677 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl"] Jul 15 19:06:07.266353 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:06:07.266326 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73198a57_057f_4510_8d27_8c9349325bc9.slice/crio-ab4f1a9c7954f93298256637e68bb77066eeef80a5ae0660082b26e65ca60846 WatchSource:0}: Error finding container ab4f1a9c7954f93298256637e68bb77066eeef80a5ae0660082b26e65ca60846: Status 404 returned error can't find the container with id ab4f1a9c7954f93298256637e68bb77066eeef80a5ae0660082b26e65ca60846 Jul 15 19:06:07.424161 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:07.424055 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" event={"ID":"73198a57-057f-4510-8d27-8c9349325bc9","Type":"ContainerStarted","Data":"ab4f1a9c7954f93298256637e68bb77066eeef80a5ae0660082b26e65ca60846"} Jul 15 19:06:07.424929 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:07.424897 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:09.460740 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.460637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:09.577802 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.576451 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs"] Jul 15 19:06:09.583562 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.583538 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:09.596307 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.596280 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4"] Jul 15 19:06:09.598406 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.598386 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 19:06:09.598508 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.598408 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jjgc8\"" Jul 15 19:06:09.601191 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.601170 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:09.605995 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.605919 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 19:06:09.606161 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.606143 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-p7s9d\"" Jul 15 19:06:09.608691 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.608670 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs"] Jul 15 19:06:09.609169 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.609073 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:09.625120 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.625002 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4"] Jul 15 19:06:09.709950 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.709901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vtw\" (UniqueName: \"kubernetes.io/projected/bd31d359-a5d0-401a-be62-3608adbe2438-kube-api-access-x8vtw\") pod \"kserve-controller-manager-d6f4b65b8-b2hqs\" (UID: \"bd31d359-a5d0-401a-be62-3608adbe2438\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:09.709950 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.709943 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rw5q\" (UniqueName: \"kubernetes.io/projected/38c6ac96-3354-47b9-ab1f-d430f23e1f25-kube-api-access-4rw5q\") pod \"llmisvc-controller-manager-777c6bd7d5-pvtl4\" (UID: \"38c6ac96-3354-47b9-ab1f-d430f23e1f25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:09.710191 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.710040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd31d359-a5d0-401a-be62-3608adbe2438-cert\") pod \"kserve-controller-manager-d6f4b65b8-b2hqs\" (UID: \"bd31d359-a5d0-401a-be62-3608adbe2438\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:09.710191 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.710089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38c6ac96-3354-47b9-ab1f-d430f23e1f25-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvtl4\" (UID: \"38c6ac96-3354-47b9-ab1f-d430f23e1f25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:09.811537 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.811505 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vtw\" (UniqueName: \"kubernetes.io/projected/bd31d359-a5d0-401a-be62-3608adbe2438-kube-api-access-x8vtw\") pod \"kserve-controller-manager-d6f4b65b8-b2hqs\" (UID: \"bd31d359-a5d0-401a-be62-3608adbe2438\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:09.811721 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.811544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rw5q\" (UniqueName: \"kubernetes.io/projected/38c6ac96-3354-47b9-ab1f-d430f23e1f25-kube-api-access-4rw5q\") pod \"llmisvc-controller-manager-777c6bd7d5-pvtl4\" (UID: \"38c6ac96-3354-47b9-ab1f-d430f23e1f25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:09.811721 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.811630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd31d359-a5d0-401a-be62-3608adbe2438-cert\") pod \"kserve-controller-manager-d6f4b65b8-b2hqs\" (UID: \"bd31d359-a5d0-401a-be62-3608adbe2438\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:09.811721 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.811661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38c6ac96-3354-47b9-ab1f-d430f23e1f25-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvtl4\" (UID: \"38c6ac96-3354-47b9-ab1f-d430f23e1f25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:09.811866 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:06:09.811816 2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 15 19:06:09.811866 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:06:09.811820 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 15 19:06:09.811961 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:06:09.811893 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd31d359-a5d0-401a-be62-3608adbe2438-cert podName:bd31d359-a5d0-401a-be62-3608adbe2438 nodeName:}" failed. No retries permitted until 2026-07-15 19:06:10.311875784 +0000 UTC m=+322.707231749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bd31d359-a5d0-401a-be62-3608adbe2438-cert") pod "kserve-controller-manager-d6f4b65b8-b2hqs" (UID: "bd31d359-a5d0-401a-be62-3608adbe2438") : secret "kserve-webhook-server-cert" not found Jul 15 19:06:09.811961 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:06:09.811908 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c6ac96-3354-47b9-ab1f-d430f23e1f25-cert podName:38c6ac96-3354-47b9-ab1f-d430f23e1f25 nodeName:}" failed. No retries permitted until 2026-07-15 19:06:10.311902353 +0000 UTC m=+322.707258302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/38c6ac96-3354-47b9-ab1f-d430f23e1f25-cert") pod "llmisvc-controller-manager-777c6bd7d5-pvtl4" (UID: "38c6ac96-3354-47b9-ab1f-d430f23e1f25") : secret "llmisvc-webhook-server-cert" not found Jul 15 19:06:09.827727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.827698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vtw\" (UniqueName: \"kubernetes.io/projected/bd31d359-a5d0-401a-be62-3608adbe2438-kube-api-access-x8vtw\") pod \"kserve-controller-manager-d6f4b65b8-b2hqs\" (UID: \"bd31d359-a5d0-401a-be62-3608adbe2438\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:09.834638 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.834616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rw5q\" (UniqueName: \"kubernetes.io/projected/38c6ac96-3354-47b9-ab1f-d430f23e1f25-kube-api-access-4rw5q\") pod \"llmisvc-controller-manager-777c6bd7d5-pvtl4\" (UID: \"38c6ac96-3354-47b9-ab1f-d430f23e1f25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:09.888525 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.888478 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:06:09.888639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.888574 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:06:09.888639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.888619 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:06:09.957121 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:09.957091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:10.211700 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.211617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:10.316403 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.316372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd31d359-a5d0-401a-be62-3608adbe2438-cert\") pod \"kserve-controller-manager-d6f4b65b8-b2hqs\" (UID: \"bd31d359-a5d0-401a-be62-3608adbe2438\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:10.316403 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.316406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38c6ac96-3354-47b9-ab1f-d430f23e1f25-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvtl4\" (UID: \"38c6ac96-3354-47b9-ab1f-d430f23e1f25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:10.318782 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.318757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd31d359-a5d0-401a-be62-3608adbe2438-cert\") pod \"kserve-controller-manager-d6f4b65b8-b2hqs\" (UID: \"bd31d359-a5d0-401a-be62-3608adbe2438\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:10.318893 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.318793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38c6ac96-3354-47b9-ab1f-d430f23e1f25-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvtl4\" (UID: \"38c6ac96-3354-47b9-ab1f-d430f23e1f25\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:10.439932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.439896 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" event={"ID":"73198a57-057f-4510-8d27-8c9349325bc9","Type":"ContainerStarted","Data":"2847bcdd646341e38700294250ab6db23532e1ebfc2f1639c9654c21fac0a6a2"} Jul 15 19:06:10.464754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.464673 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" podStartSLOduration=1.844988631 podStartE2EDuration="4.464657965s" totalImagesPullingTime="2.619669334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:10.463279055 +0000 UTC m=+322.858635026" watchObservedRunningTime="2026-07-15 19:06:10.464657965 +0000 UTC m=+322.860013935" Jul 15 19:06:10.497385 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.497360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:10.514296 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.514274 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:10.643838 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.643815 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs"] Jul 15 19:06:10.645670 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:06:10.645647 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd31d359_a5d0_401a_be62_3608adbe2438.slice/crio-05574c5f89caad4616c8584d611fde2c0da16f8abdca315ce9b001b6abe1d4b8 WatchSource:0}: Error finding container 05574c5f89caad4616c8584d611fde2c0da16f8abdca315ce9b001b6abe1d4b8: Status 404 returned error can't find the container with id 05574c5f89caad4616c8584d611fde2c0da16f8abdca315ce9b001b6abe1d4b8 Jul 15 19:06:10.677427 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.677404 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4"] Jul 15 19:06:10.681429 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:06:10.681406 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38c6ac96_3354_47b9_ab1f_d430f23e1f25.slice/crio-430339ad8697b65e29e8a89d3fd384b338dd0a7c35340964363f079fb511bed6 WatchSource:0}: Error finding container 430339ad8697b65e29e8a89d3fd384b338dd0a7c35340964363f079fb511bed6: Status 404 returned error can't find the container with id 430339ad8697b65e29e8a89d3fd384b338dd0a7c35340964363f079fb511bed6 Jul 15 19:06:10.709889 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.709866 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rptvn"] Jul 15 19:06:10.710094 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.710064 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" podUID="e54ee25d-9f31-41ab-a573-5de73fbb03d9" containerName="discovery" containerID="cri-o://ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7" gracePeriod=30 Jul 15 19:06:10.906298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.906259 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:10.946005 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:10.945983 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:11.018353 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.018316 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:11.023231 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023212 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-ca-configmap\") pod \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " Jul 15 19:06:11.023319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023270 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-token\") pod \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " Jul 15 19:06:11.023319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023302 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-cacerts\") pod \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " Jul 15 19:06:11.023384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023330 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ee25d-9f31-41ab-a573-5de73fbb03d9-local-certs\") pod \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " Jul 15 19:06:11.023384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023364 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-dns-cert\") pod \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " Jul 15 19:06:11.023510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023392 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sskkm\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-kube-api-access-sskkm\") pod \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " Jul 15 19:06:11.023510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023427 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-kubeconfig\") pod \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\" (UID: \"e54ee25d-9f31-41ab-a573-5de73fbb03d9\") " Jul 15 19:06:11.023661 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.023635 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e54ee25d-9f31-41ab-a573-5de73fbb03d9" (UID: "e54ee25d-9f31-41ab-a573-5de73fbb03d9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:06:11.025701 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.025676 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54ee25d-9f31-41ab-a573-5de73fbb03d9-local-certs" (OuterVolumeSpecName: "local-certs") pod "e54ee25d-9f31-41ab-a573-5de73fbb03d9" (UID: "e54ee25d-9f31-41ab-a573-5de73fbb03d9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:06:11.025812 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.025696 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-kube-api-access-sskkm" (OuterVolumeSpecName: "kube-api-access-sskkm") pod "e54ee25d-9f31-41ab-a573-5de73fbb03d9" (UID: "e54ee25d-9f31-41ab-a573-5de73fbb03d9"). InnerVolumeSpecName "kube-api-access-sskkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:06:11.025934 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.025918 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "e54ee25d-9f31-41ab-a573-5de73fbb03d9" (UID: "e54ee25d-9f31-41ab-a573-5de73fbb03d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:06:11.025993 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.025947 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-cacerts" (OuterVolumeSpecName: "cacerts") pod "e54ee25d-9f31-41ab-a573-5de73fbb03d9" (UID: "e54ee25d-9f31-41ab-a573-5de73fbb03d9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:06:11.026037 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.025988 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e54ee25d-9f31-41ab-a573-5de73fbb03d9" (UID: "e54ee25d-9f31-41ab-a573-5de73fbb03d9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:06:11.026091 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.026067 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e54ee25d-9f31-41ab-a573-5de73fbb03d9" (UID: "e54ee25d-9f31-41ab-a573-5de73fbb03d9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:06:11.124658 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.124621 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-ca-configmap\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:06:11.124658 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.124653 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:06:11.124658 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.124667 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-cacerts\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:06:11.124894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.124678 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ee25d-9f31-41ab-a573-5de73fbb03d9-local-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:06:11.124894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.124688 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-csr-dns-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:06:11.124894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.124700 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sskkm\" (UniqueName: \"kubernetes.io/projected/e54ee25d-9f31-41ab-a573-5de73fbb03d9-kube-api-access-sskkm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:06:11.124894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.124711 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54ee25d-9f31-41ab-a573-5de73fbb03d9-istio-kubeconfig\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:06:11.130608 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.130588 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:11.134906 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.134884 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:11.446225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.446136 2582 generic.go:356] "Generic (PLEG): container finished" podID="e54ee25d-9f31-41ab-a573-5de73fbb03d9" containerID="ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7" exitCode=0 Jul 15 19:06:11.446225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.446180 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" Jul 15 19:06:11.446225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.446216 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" event={"ID":"e54ee25d-9f31-41ab-a573-5de73fbb03d9","Type":"ContainerDied","Data":"ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7"} Jul 15 19:06:11.446489 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.446277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rptvn" event={"ID":"e54ee25d-9f31-41ab-a573-5de73fbb03d9","Type":"ContainerDied","Data":"a3de29d58c215b86e0b85375b66a1d1de629374b7c35b1fb3f91aecce5c4bd58"} Jul 15 19:06:11.446489 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.446298 2582 scope.go:117] "RemoveContainer" containerID="ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7" Jul 15 19:06:11.447677 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.447652 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" event={"ID":"bd31d359-a5d0-401a-be62-3608adbe2438","Type":"ContainerStarted","Data":"05574c5f89caad4616c8584d611fde2c0da16f8abdca315ce9b001b6abe1d4b8"} Jul 15 19:06:11.448764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.448743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" event={"ID":"38c6ac96-3354-47b9-ab1f-d430f23e1f25","Type":"ContainerStarted","Data":"430339ad8697b65e29e8a89d3fd384b338dd0a7c35340964363f079fb511bed6"} Jul 15 19:06:11.449068 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.449052 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:11.449868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.449851 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j9hcl" Jul 15 19:06:11.455197 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.455181 2582 scope.go:117] "RemoveContainer" containerID="ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7" Jul 15 19:06:11.455458 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:06:11.455439 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7\": container with ID starting with ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7 not found: ID does not exist" containerID="ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7" Jul 15 19:06:11.455518 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.455466 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7"} err="failed to get container status \"ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7\": rpc error: code = NotFound desc = could not find container \"ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7\": container with ID starting with ce34a373833aa0abb3c60c65bc555827afcf75464b723566fb0cd327a2159ef7 not found: ID does not exist" Jul 15 19:06:11.532511 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.532481 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rptvn"] Jul 15 19:06:11.548293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:11.548269 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rptvn"] Jul 15 19:06:12.207606 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:12.207573 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54ee25d-9f31-41ab-a573-5de73fbb03d9" path="/var/lib/kubelet/pods/e54ee25d-9f31-41ab-a573-5de73fbb03d9/volumes" Jul 15 19:06:12.999692 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:12.999544 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:13.247155 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:13.247128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:13.462763 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:13.462718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" event={"ID":"bd31d359-a5d0-401a-be62-3608adbe2438","Type":"ContainerStarted","Data":"16f8a61f5ddc613c08325c8882a5c3b9e8f7075d1fc00614e261ead23dc73764"} Jul 15 19:06:13.462965 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:13.462946 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:13.520919 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:13.520798 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" podStartSLOduration=1.956079548 podStartE2EDuration="4.520781538s" totalImagesPullingTime="2.56470199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:13.520471556 +0000 UTC m=+325.915827525" watchObservedRunningTime="2026-07-15 19:06:13.520781538 +0000 UTC m=+325.916137507" Jul 15 19:06:14.467962 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:14.467926 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" event={"ID":"38c6ac96-3354-47b9-ab1f-d430f23e1f25","Type":"ContainerStarted","Data":"01caf2f4eb1f673008f8a8debac2728a98f09e21911a729122f1e9b501ef2e11"} Jul 15 19:06:14.468358 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:14.468096 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:14.488195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:14.488143 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" podStartSLOduration=1.8412755079999998 podStartE2EDuration="5.488127571s" totalImagesPullingTime="3.646852063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:14.486724658 +0000 UTC m=+326.882080651" watchObservedRunningTime="2026-07-15 19:06:14.488127571 +0000 UTC m=+326.883483540" Jul 15 19:06:17.428303 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:17.428273 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-slhht" Jul 15 19:06:18.430912 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:18.430883 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7gvhd" Jul 15 19:06:44.473204 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:44.473170 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b2hqs" Jul 15 19:06:45.474383 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:45.474351 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvtl4" Jul 15 19:06:59.548665 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.548624 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:59.645112 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.645077 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:59.674448 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.674412 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-shthj"] Jul 15 19:06:59.674737 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.674724 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54ee25d-9f31-41ab-a573-5de73fbb03d9" containerName="discovery" Jul 15 19:06:59.674779 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.674738 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54ee25d-9f31-41ab-a573-5de73fbb03d9" containerName="discovery" Jul 15 19:06:59.674812 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.674791 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54ee25d-9f31-41ab-a573-5de73fbb03d9" containerName="discovery" Jul 15 19:06:59.682038 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.682014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-shthj" Jul 15 19:06:59.684966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.684941 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-g946r\"" Jul 15 19:06:59.685308 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.685294 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:06:59.685597 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.685582 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:06:59.690997 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.690970 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-shthj"] Jul 15 19:06:59.737620 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.737589 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsd9g\" (UniqueName: \"kubernetes.io/projected/b6431b26-22f1-4d8e-80cc-6125d29ff157-kube-api-access-gsd9g\") pod \"kuadrant-operator-catalog-shthj\" (UID: \"b6431b26-22f1-4d8e-80cc-6125d29ff157\") " pod="kuadrant-system/kuadrant-operator-catalog-shthj" Jul 15 19:06:59.746978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.746945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:06:59.838298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.838195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsd9g\" (UniqueName: \"kubernetes.io/projected/b6431b26-22f1-4d8e-80cc-6125d29ff157-kube-api-access-gsd9g\") pod \"kuadrant-operator-catalog-shthj\" (UID: \"b6431b26-22f1-4d8e-80cc-6125d29ff157\") " pod="kuadrant-system/kuadrant-operator-catalog-shthj" Jul 15 19:06:59.848981 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.848957 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsd9g\" (UniqueName: \"kubernetes.io/projected/b6431b26-22f1-4d8e-80cc-6125d29ff157-kube-api-access-gsd9g\") pod \"kuadrant-operator-catalog-shthj\" (UID: \"b6431b26-22f1-4d8e-80cc-6125d29ff157\") " pod="kuadrant-system/kuadrant-operator-catalog-shthj" Jul 15 19:06:59.994146 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:06:59.994094 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-shthj" Jul 15 19:07:00.027425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.027393 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-shthj"] Jul 15 19:07:00.128635 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.128611 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-shthj"] Jul 15 19:07:00.130419 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:00.130392 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6431b26_22f1_4d8e_80cc_6125d29ff157.slice/crio-837cf196cd6d354cb93c08bce520b8cdb848cae79f7682072928b768e05e3e4f WatchSource:0}: Error finding container 837cf196cd6d354cb93c08bce520b8cdb848cae79f7682072928b768e05e3e4f: Status 404 returned error can't find the container with id 837cf196cd6d354cb93c08bce520b8cdb848cae79f7682072928b768e05e3e4f Jul 15 19:07:00.240447 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.240416 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g4tzg"] Jul 15 19:07:00.243267 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.243229 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:00.251618 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.251585 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g4tzg"] Jul 15 19:07:00.342936 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.342902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tllvp\" (UniqueName: \"kubernetes.io/projected/e51f9177-f90e-4920-8fee-8c651f54ddbf-kube-api-access-tllvp\") pod \"kuadrant-operator-catalog-g4tzg\" (UID: \"e51f9177-f90e-4920-8fee-8c651f54ddbf\") " pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:00.444235 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.444144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllvp\" (UniqueName: \"kubernetes.io/projected/e51f9177-f90e-4920-8fee-8c651f54ddbf-kube-api-access-tllvp\") pod \"kuadrant-operator-catalog-g4tzg\" (UID: \"e51f9177-f90e-4920-8fee-8c651f54ddbf\") " pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:00.452816 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.452787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllvp\" (UniqueName: \"kubernetes.io/projected/e51f9177-f90e-4920-8fee-8c651f54ddbf-kube-api-access-tllvp\") pod \"kuadrant-operator-catalog-g4tzg\" (UID: \"e51f9177-f90e-4920-8fee-8c651f54ddbf\") " pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:00.553211 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.553172 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:00.664460 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.664420 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-shthj" event={"ID":"b6431b26-22f1-4d8e-80cc-6125d29ff157","Type":"ContainerStarted","Data":"837cf196cd6d354cb93c08bce520b8cdb848cae79f7682072928b768e05e3e4f"} Jul 15 19:07:00.688716 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:00.688603 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g4tzg"] Jul 15 19:07:00.703215 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:00.703144 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51f9177_f90e_4920_8fee_8c651f54ddbf.slice/crio-38957acba459e24dfc4fbddf8c66c8f0e77ee868f03bbbd5237c2819c2abd135 WatchSource:0}: Error finding container 38957acba459e24dfc4fbddf8c66c8f0e77ee868f03bbbd5237c2819c2abd135: Status 404 returned error can't find the container with id 38957acba459e24dfc4fbddf8c66c8f0e77ee868f03bbbd5237c2819c2abd135 Jul 15 19:07:01.671582 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:01.671540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" event={"ID":"e51f9177-f90e-4920-8fee-8c651f54ddbf","Type":"ContainerStarted","Data":"38957acba459e24dfc4fbddf8c66c8f0e77ee868f03bbbd5237c2819c2abd135"} Jul 15 19:07:02.677127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.677078 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-shthj" event={"ID":"b6431b26-22f1-4d8e-80cc-6125d29ff157","Type":"ContainerStarted","Data":"b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970"} Jul 15 19:07:02.677127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.677086 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-shthj" podUID="b6431b26-22f1-4d8e-80cc-6125d29ff157" containerName="registry-server" containerID="cri-o://b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970" gracePeriod=2 Jul 15 19:07:02.678638 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.678602 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" event={"ID":"e51f9177-f90e-4920-8fee-8c651f54ddbf","Type":"ContainerStarted","Data":"7289f76460feb6505c650590bdfab36ab2afbea1df364cc919ac68665a3ca347"} Jul 15 19:07:02.695170 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.695114 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-shthj" podStartSLOduration=1.730577599 podStartE2EDuration="3.695100583s" totalImagesPullingTime="1.964522984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:07:02.693367727 +0000 UTC m=+375.088723696" watchObservedRunningTime="2026-07-15 19:07:02.695100583 +0000 UTC m=+375.090456552" Jul 15 19:07:02.712300 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.712260 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" podStartSLOduration=1.317186198 podStartE2EDuration="2.712234242s" totalImagesPullingTime="1.395048044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:07:02.710923866 +0000 UTC m=+375.106279837" watchObservedRunningTime="2026-07-15 19:07:02.712234242 +0000 UTC m=+375.107590211" Jul 15 19:07:02.915745 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.915721 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-shthj" Jul 15 19:07:02.967115 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.967030 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsd9g\" (UniqueName: \"kubernetes.io/projected/b6431b26-22f1-4d8e-80cc-6125d29ff157-kube-api-access-gsd9g\") pod \"b6431b26-22f1-4d8e-80cc-6125d29ff157\" (UID: \"b6431b26-22f1-4d8e-80cc-6125d29ff157\") " Jul 15 19:07:02.969201 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:02.969176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6431b26-22f1-4d8e-80cc-6125d29ff157-kube-api-access-gsd9g" (OuterVolumeSpecName: "kube-api-access-gsd9g") pod "b6431b26-22f1-4d8e-80cc-6125d29ff157" (UID: "b6431b26-22f1-4d8e-80cc-6125d29ff157"). InnerVolumeSpecName "kube-api-access-gsd9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:07:03.067881 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.067848 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsd9g\" (UniqueName: \"kubernetes.io/projected/b6431b26-22f1-4d8e-80cc-6125d29ff157-kube-api-access-gsd9g\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:03.685305 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.685262 2582 generic.go:356] "Generic (PLEG): container finished" podID="b6431b26-22f1-4d8e-80cc-6125d29ff157" containerID="b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970" exitCode=0 Jul 15 19:07:03.685305 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.685304 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-shthj" Jul 15 19:07:03.685885 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.685332 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-shthj" event={"ID":"b6431b26-22f1-4d8e-80cc-6125d29ff157","Type":"ContainerDied","Data":"b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970"} Jul 15 19:07:03.685885 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.685388 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-shthj" event={"ID":"b6431b26-22f1-4d8e-80cc-6125d29ff157","Type":"ContainerDied","Data":"837cf196cd6d354cb93c08bce520b8cdb848cae79f7682072928b768e05e3e4f"} Jul 15 19:07:03.685885 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.685416 2582 scope.go:117] "RemoveContainer" containerID="b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970" Jul 15 19:07:03.694209 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.694193 2582 scope.go:117] "RemoveContainer" containerID="b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970" Jul 15 19:07:03.694552 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:07:03.694531 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970\": container with ID starting with b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970 not found: ID does not exist" containerID="b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970" Jul 15 19:07:03.694646 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.694560 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970"} err="failed to get container status \"b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970\": rpc error: code = NotFound desc = could not find container \"b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970\": container with ID starting with b2c030b2bfff25c00507cf13620b286b7a2e2bb3db88c632da1cb86764c80970 not found: ID does not exist" Jul 15 19:07:03.705888 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.705864 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-shthj"] Jul 15 19:07:03.712555 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:03.712537 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-shthj"] Jul 15 19:07:04.204129 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:04.204097 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6431b26-22f1-4d8e-80cc-6125d29ff157" path="/var/lib/kubelet/pods/b6431b26-22f1-4d8e-80cc-6125d29ff157/volumes" Jul 15 19:07:10.553708 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:10.553666 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:10.554179 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:10.553747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:10.575464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:10.575432 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:10.737999 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:10.737973 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-g4tzg" Jul 15 19:07:15.028566 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.028532 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq"] Jul 15 19:07:15.029029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.028962 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6431b26-22f1-4d8e-80cc-6125d29ff157" containerName="registry-server" Jul 15 19:07:15.029029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.028980 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6431b26-22f1-4d8e-80cc-6125d29ff157" containerName="registry-server" Jul 15 19:07:15.029152 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.029079 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6431b26-22f1-4d8e-80cc-6125d29ff157" containerName="registry-server" Jul 15 19:07:15.031124 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.031103 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.034436 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.034408 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n6qmk\"" Jul 15 19:07:15.043291 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.043271 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq"] Jul 15 19:07:15.166515 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.166479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.166515 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.166515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.166746 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.166540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59n92\" (UniqueName: \"kubernetes.io/projected/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-kube-api-access-59n92\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.267224 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.267189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.267442 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.267267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59n92\" (UniqueName: \"kubernetes.io/projected/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-kube-api-access-59n92\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.267442 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.267431 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.267668 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.267630 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.267745 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.267724 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.278656 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.278603 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59n92\" (UniqueName: \"kubernetes.io/projected/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-kube-api-access-59n92\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.340231 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.340194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:15.467395 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.467365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq"] Jul 15 19:07:15.469660 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:15.469628 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c2907f_e2d3_4bd4_82c4_a22bcd6a7a71.slice/crio-50cae55732c049a1e018cc43e986378563a93bca1cb5e64373f3eaafb256387b WatchSource:0}: Error finding container 50cae55732c049a1e018cc43e986378563a93bca1cb5e64373f3eaafb256387b: Status 404 returned error can't find the container with id 50cae55732c049a1e018cc43e986378563a93bca1cb5e64373f3eaafb256387b Jul 15 19:07:15.619233 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.619203 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g"] Jul 15 19:07:15.621554 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.621539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.631809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.631788 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g"] Jul 15 19:07:15.737515 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.737477 2582 generic.go:356] "Generic (PLEG): container finished" podID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerID="a51106749d559548e7d332d40b33cacd9fe29cc5ea649675ae284e947e39b3ad" exitCode=0 Jul 15 19:07:15.737679 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.737564 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" event={"ID":"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71","Type":"ContainerDied","Data":"a51106749d559548e7d332d40b33cacd9fe29cc5ea649675ae284e947e39b3ad"} Jul 15 19:07:15.737679 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.737603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" event={"ID":"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71","Type":"ContainerStarted","Data":"50cae55732c049a1e018cc43e986378563a93bca1cb5e64373f3eaafb256387b"} Jul 15 19:07:15.772465 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.772430 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vgw\" (UniqueName: \"kubernetes.io/projected/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-kube-api-access-l7vgw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.772593 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.772477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.772593 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.772541 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.873927 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.873845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.873927 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.873921 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vgw\" (UniqueName: \"kubernetes.io/projected/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-kube-api-access-l7vgw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.874089 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.873947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.874349 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.874333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.874390 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.874335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.883670 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.883640 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vgw\" (UniqueName: \"kubernetes.io/projected/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-kube-api-access-l7vgw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:15.929798 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:15.929768 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:16.054759 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.054605 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g"] Jul 15 19:07:16.056838 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:16.056806 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae4218b_87c0_488e_b2fc_1d3052adcc1d.slice/crio-5d4f9a0aab7b2df99dc058b75e608b7636c1ae62b7d156ae1e2ff37ca639eacd WatchSource:0}: Error finding container 5d4f9a0aab7b2df99dc058b75e608b7636c1ae62b7d156ae1e2ff37ca639eacd: Status 404 returned error can't find the container with id 5d4f9a0aab7b2df99dc058b75e608b7636c1ae62b7d156ae1e2ff37ca639eacd Jul 15 19:07:16.218956 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.218924 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np"] Jul 15 19:07:16.221264 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.221229 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.240214 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.240191 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np"] Jul 15 19:07:16.378634 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.378596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.378803 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.378651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjvx\" (UniqueName: \"kubernetes.io/projected/5e57dcbb-ff75-49cd-b516-260e548ea9ab-kube-api-access-mbjvx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.378803 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.378731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.479563 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.479530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.479697 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.479584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.479697 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.479616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjvx\" (UniqueName: \"kubernetes.io/projected/5e57dcbb-ff75-49cd-b516-260e548ea9ab-kube-api-access-mbjvx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.480000 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.479974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.480037 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.479988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.491971 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.491941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjvx\" (UniqueName: \"kubernetes.io/projected/5e57dcbb-ff75-49cd-b516-260e548ea9ab-kube-api-access-mbjvx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.530061 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.530026 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:16.639743 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.639711 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp"] Jul 15 19:07:16.642841 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.642822 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.654867 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.654843 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp"] Jul 15 19:07:16.661936 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.661914 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np"] Jul 15 19:07:16.665092 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:16.665065 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e57dcbb_ff75_49cd_b516_260e548ea9ab.slice/crio-f72c54777e46e7745a9595ffe03ee4ccafc1fbe6011d735f85e526ccb797cb71 WatchSource:0}: Error finding container f72c54777e46e7745a9595ffe03ee4ccafc1fbe6011d735f85e526ccb797cb71: Status 404 returned error can't find the container with id f72c54777e46e7745a9595ffe03ee4ccafc1fbe6011d735f85e526ccb797cb71 Jul 15 19:07:16.743964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.743865 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" event={"ID":"5e57dcbb-ff75-49cd-b516-260e548ea9ab","Type":"ContainerStarted","Data":"094227c8882e16f33065789215244d1c8b67245b6e85115597d0cafce2849dbd"} Jul 15 19:07:16.743964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.743928 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" event={"ID":"5e57dcbb-ff75-49cd-b516-260e548ea9ab","Type":"ContainerStarted","Data":"f72c54777e46e7745a9595ffe03ee4ccafc1fbe6011d735f85e526ccb797cb71"} Jul 15 19:07:16.745466 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.745438 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerID="4ee4124aafc24c8254805d88258fd01d4273f7ed1ca63e13a4e7fb0822162f59" exitCode=0 Jul 15 19:07:16.745587 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.745467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" event={"ID":"6ae4218b-87c0-488e-b2fc-1d3052adcc1d","Type":"ContainerDied","Data":"4ee4124aafc24c8254805d88258fd01d4273f7ed1ca63e13a4e7fb0822162f59"} Jul 15 19:07:16.745587 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.745513 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" event={"ID":"6ae4218b-87c0-488e-b2fc-1d3052adcc1d","Type":"ContainerStarted","Data":"5d4f9a0aab7b2df99dc058b75e608b7636c1ae62b7d156ae1e2ff37ca639eacd"} Jul 15 19:07:16.781759 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.781730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.781873 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.781785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvwj\" (UniqueName: \"kubernetes.io/projected/433b012f-7bc3-4e6e-ac9f-436899404f23-kube-api-access-6xvwj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.781873 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.781836 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.882666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.882630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.882847 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.882696 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvwj\" (UniqueName: \"kubernetes.io/projected/433b012f-7bc3-4e6e-ac9f-436899404f23-kube-api-access-6xvwj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.882847 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.882731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.883003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.882984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.883115 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.883096 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.929542 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.929510 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvwj\" (UniqueName: \"kubernetes.io/projected/433b012f-7bc3-4e6e-ac9f-436899404f23-kube-api-access-6xvwj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:16.951879 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:16.951850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:17.082916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.082886 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp"] Jul 15 19:07:17.083782 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:17.083763 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433b012f_7bc3_4e6e_ac9f_436899404f23.slice/crio-d1fedb1922707662c42cb0b9ba70f2ca80f715d0c6ea652b07007bba7e55ba96 WatchSource:0}: Error finding container d1fedb1922707662c42cb0b9ba70f2ca80f715d0c6ea652b07007bba7e55ba96: Status 404 returned error can't find the container with id d1fedb1922707662c42cb0b9ba70f2ca80f715d0c6ea652b07007bba7e55ba96 Jul 15 19:07:17.753025 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.752998 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerID="9407bb2c0e819fb04191eb9d13218a72ddf60aa1103231fbac05f89e1c9dc2ea" exitCode=0 Jul 15 19:07:17.753189 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.753085 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" event={"ID":"6ae4218b-87c0-488e-b2fc-1d3052adcc1d","Type":"ContainerDied","Data":"9407bb2c0e819fb04191eb9d13218a72ddf60aa1103231fbac05f89e1c9dc2ea"} Jul 15 19:07:17.754869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.754848 2582 generic.go:356] "Generic (PLEG): container finished" podID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerID="b2f02f3c1c95b5451d91537fe5fc579d52e0dd9291d40a7d8abb63205c5c25af" exitCode=0 Jul 15 19:07:17.754935 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.754904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" event={"ID":"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71","Type":"ContainerDied","Data":"b2f02f3c1c95b5451d91537fe5fc579d52e0dd9291d40a7d8abb63205c5c25af"} Jul 15 19:07:17.756486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.756443 2582 generic.go:356] "Generic (PLEG): container finished" podID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerID="b67baec1e1f4f9a817c3f73a4986be8efea8a44dd6c12dc02738f27073d90959" exitCode=0 Jul 15 19:07:17.756558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.756524 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" event={"ID":"433b012f-7bc3-4e6e-ac9f-436899404f23","Type":"ContainerDied","Data":"b67baec1e1f4f9a817c3f73a4986be8efea8a44dd6c12dc02738f27073d90959"} Jul 15 19:07:17.756612 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.756563 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" event={"ID":"433b012f-7bc3-4e6e-ac9f-436899404f23","Type":"ContainerStarted","Data":"d1fedb1922707662c42cb0b9ba70f2ca80f715d0c6ea652b07007bba7e55ba96"} Jul 15 19:07:17.758122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.758099 2582 generic.go:356] "Generic (PLEG): container finished" podID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerID="094227c8882e16f33065789215244d1c8b67245b6e85115597d0cafce2849dbd" exitCode=0 Jul 15 19:07:17.758259 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:17.758131 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" event={"ID":"5e57dcbb-ff75-49cd-b516-260e548ea9ab","Type":"ContainerDied","Data":"094227c8882e16f33065789215244d1c8b67245b6e85115597d0cafce2849dbd"} Jul 15 19:07:18.765870 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:18.765840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" event={"ID":"5e57dcbb-ff75-49cd-b516-260e548ea9ab","Type":"ContainerStarted","Data":"5c6a540198fc4251b451e32fe45566797d88528215d29ada1f26c1ebc16316f5"} Jul 15 19:07:18.768001 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:18.767978 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerID="9da963ad8d7b25f66db95196fd55e78205b5d8c655f16f99cb329c56f05c258d" exitCode=0 Jul 15 19:07:18.768112 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:18.768034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" event={"ID":"6ae4218b-87c0-488e-b2fc-1d3052adcc1d","Type":"ContainerDied","Data":"9da963ad8d7b25f66db95196fd55e78205b5d8c655f16f99cb329c56f05c258d"} Jul 15 19:07:18.770289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:18.770266 2582 generic.go:356] "Generic (PLEG): container finished" podID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerID="5740daee88a4c7ee6c172eea214d5075c000adca0af6c6c4935dbc94840c6e95" exitCode=0 Jul 15 19:07:18.770394 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:18.770332 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" event={"ID":"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71","Type":"ContainerDied","Data":"5740daee88a4c7ee6c172eea214d5075c000adca0af6c6c4935dbc94840c6e95"} Jul 15 19:07:19.777872 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:19.777842 2582 generic.go:356] "Generic (PLEG): container finished" podID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerID="b484d0ad5bcfd7d8cb8cd59cbc5faac3cbde285d0921224d6fc5881c7c6ebbf4" exitCode=0 Jul 15 19:07:19.778365 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:19.777914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" event={"ID":"433b012f-7bc3-4e6e-ac9f-436899404f23","Type":"ContainerDied","Data":"b484d0ad5bcfd7d8cb8cd59cbc5faac3cbde285d0921224d6fc5881c7c6ebbf4"} Jul 15 19:07:19.779814 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:19.779786 2582 generic.go:356] "Generic (PLEG): container finished" podID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerID="5c6a540198fc4251b451e32fe45566797d88528215d29ada1f26c1ebc16316f5" exitCode=0 Jul 15 19:07:19.779893 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:19.779837 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" event={"ID":"5e57dcbb-ff75-49cd-b516-260e548ea9ab","Type":"ContainerDied","Data":"5c6a540198fc4251b451e32fe45566797d88528215d29ada1f26c1ebc16316f5"} Jul 15 19:07:19.930113 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:19.930091 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:20.014182 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.014156 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-util\") pod \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " Jul 15 19:07:20.014356 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.014222 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7vgw\" (UniqueName: \"kubernetes.io/projected/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-kube-api-access-l7vgw\") pod \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " Jul 15 19:07:20.014356 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.014317 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-bundle\") pod \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\" (UID: \"6ae4218b-87c0-488e-b2fc-1d3052adcc1d\") " Jul 15 19:07:20.014792 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.014767 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-bundle" (OuterVolumeSpecName: "bundle") pod "6ae4218b-87c0-488e-b2fc-1d3052adcc1d" (UID: "6ae4218b-87c0-488e-b2fc-1d3052adcc1d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:20.016357 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.016335 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-kube-api-access-l7vgw" (OuterVolumeSpecName: "kube-api-access-l7vgw") pod "6ae4218b-87c0-488e-b2fc-1d3052adcc1d" (UID: "6ae4218b-87c0-488e-b2fc-1d3052adcc1d"). InnerVolumeSpecName "kube-api-access-l7vgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:07:20.019528 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.019507 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-util" (OuterVolumeSpecName: "util") pod "6ae4218b-87c0-488e-b2fc-1d3052adcc1d" (UID: "6ae4218b-87c0-488e-b2fc-1d3052adcc1d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:20.034693 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.034637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:20.115168 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.115135 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-util\") pod \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " Jul 15 19:07:20.115346 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.115175 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-bundle\") pod \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " Jul 15 19:07:20.115346 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.115212 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59n92\" (UniqueName: \"kubernetes.io/projected/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-kube-api-access-59n92\") pod \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\" (UID: \"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71\") " Jul 15 19:07:20.115485 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.115377 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7vgw\" (UniqueName: \"kubernetes.io/projected/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-kube-api-access-l7vgw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:20.115485 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.115390 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:20.115485 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.115398 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae4218b-87c0-488e-b2fc-1d3052adcc1d-util\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:20.115809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.115777 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-bundle" (OuterVolumeSpecName: "bundle") pod "a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" (UID: "a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:20.117264 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.117227 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-kube-api-access-59n92" (OuterVolumeSpecName: "kube-api-access-59n92") pod "a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" (UID: "a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71"). InnerVolumeSpecName "kube-api-access-59n92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:07:20.121054 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.121024 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-util" (OuterVolumeSpecName: "util") pod "a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" (UID: "a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:20.216450 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.216428 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-util\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:20.216450 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.216449 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:20.216580 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.216458 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59n92\" (UniqueName: \"kubernetes.io/projected/a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71-kube-api-access-59n92\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:20.787025 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.786986 2582 generic.go:356] "Generic (PLEG): container finished" podID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerID="c7d3eefd438fbab5d9fd576c3bf197b1ae0b8fed3b22823337033ac4e2a357ac" exitCode=0 Jul 15 19:07:20.787461 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.787079 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" event={"ID":"5e57dcbb-ff75-49cd-b516-260e548ea9ab","Type":"ContainerDied","Data":"c7d3eefd438fbab5d9fd576c3bf197b1ae0b8fed3b22823337033ac4e2a357ac"} Jul 15 19:07:20.788862 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.788845 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" Jul 15 19:07:20.788975 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.788863 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g" event={"ID":"6ae4218b-87c0-488e-b2fc-1d3052adcc1d","Type":"ContainerDied","Data":"5d4f9a0aab7b2df99dc058b75e608b7636c1ae62b7d156ae1e2ff37ca639eacd"} Jul 15 19:07:20.788975 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.788900 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d4f9a0aab7b2df99dc058b75e608b7636c1ae62b7d156ae1e2ff37ca639eacd" Jul 15 19:07:20.790650 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.790594 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" event={"ID":"a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71","Type":"ContainerDied","Data":"50cae55732c049a1e018cc43e986378563a93bca1cb5e64373f3eaafb256387b"} Jul 15 19:07:20.790809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.790665 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50cae55732c049a1e018cc43e986378563a93bca1cb5e64373f3eaafb256387b" Jul 15 19:07:20.790809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.790605 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq" Jul 15 19:07:20.793190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.793172 2582 generic.go:356] "Generic (PLEG): container finished" podID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerID="5e152f48a004b566aaae4a8a1f54eb855a05d0572817cfa0654b11856617ccd5" exitCode=0 Jul 15 19:07:20.793271 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:20.793197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" event={"ID":"433b012f-7bc3-4e6e-ac9f-436899404f23","Type":"ContainerDied","Data":"5e152f48a004b566aaae4a8a1f54eb855a05d0572817cfa0654b11856617ccd5"} Jul 15 19:07:21.953669 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:21.953643 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:21.956681 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:21.956657 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:22.032491 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.032463 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-bundle\") pod \"433b012f-7bc3-4e6e-ac9f-436899404f23\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " Jul 15 19:07:22.032676 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.032506 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-bundle\") pod \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " Jul 15 19:07:22.032676 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.032535 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-util\") pod \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " Jul 15 19:07:22.032676 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.032573 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbjvx\" (UniqueName: \"kubernetes.io/projected/5e57dcbb-ff75-49cd-b516-260e548ea9ab-kube-api-access-mbjvx\") pod \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\" (UID: \"5e57dcbb-ff75-49cd-b516-260e548ea9ab\") " Jul 15 19:07:22.032676 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.032639 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-util\") pod \"433b012f-7bc3-4e6e-ac9f-436899404f23\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " Jul 15 19:07:22.032676 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.032671 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xvwj\" (UniqueName: \"kubernetes.io/projected/433b012f-7bc3-4e6e-ac9f-436899404f23-kube-api-access-6xvwj\") pod \"433b012f-7bc3-4e6e-ac9f-436899404f23\" (UID: \"433b012f-7bc3-4e6e-ac9f-436899404f23\") " Jul 15 19:07:22.033079 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.033045 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-bundle" (OuterVolumeSpecName: "bundle") pod "433b012f-7bc3-4e6e-ac9f-436899404f23" (UID: "433b012f-7bc3-4e6e-ac9f-436899404f23"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:22.033201 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.033140 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-bundle" (OuterVolumeSpecName: "bundle") pod "5e57dcbb-ff75-49cd-b516-260e548ea9ab" (UID: "5e57dcbb-ff75-49cd-b516-260e548ea9ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:22.034809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.034787 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e57dcbb-ff75-49cd-b516-260e548ea9ab-kube-api-access-mbjvx" (OuterVolumeSpecName: "kube-api-access-mbjvx") pod "5e57dcbb-ff75-49cd-b516-260e548ea9ab" (UID: "5e57dcbb-ff75-49cd-b516-260e548ea9ab"). InnerVolumeSpecName "kube-api-access-mbjvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:07:22.035218 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.035196 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433b012f-7bc3-4e6e-ac9f-436899404f23-kube-api-access-6xvwj" (OuterVolumeSpecName: "kube-api-access-6xvwj") pod "433b012f-7bc3-4e6e-ac9f-436899404f23" (UID: "433b012f-7bc3-4e6e-ac9f-436899404f23"). InnerVolumeSpecName "kube-api-access-6xvwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:07:22.039561 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.039537 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-util" (OuterVolumeSpecName: "util") pod "433b012f-7bc3-4e6e-ac9f-436899404f23" (UID: "433b012f-7bc3-4e6e-ac9f-436899404f23"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:22.039921 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.039899 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-util" (OuterVolumeSpecName: "util") pod "5e57dcbb-ff75-49cd-b516-260e548ea9ab" (UID: "5e57dcbb-ff75-49cd-b516-260e548ea9ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:07:22.133945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.133862 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xvwj\" (UniqueName: \"kubernetes.io/projected/433b012f-7bc3-4e6e-ac9f-436899404f23-kube-api-access-6xvwj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:22.133945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.133889 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:22.133945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.133902 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:22.133945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.133914 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e57dcbb-ff75-49cd-b516-260e548ea9ab-util\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:22.133945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.133926 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjvx\" (UniqueName: \"kubernetes.io/projected/5e57dcbb-ff75-49cd-b516-260e548ea9ab-kube-api-access-mbjvx\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:22.133945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.133938 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/433b012f-7bc3-4e6e-ac9f-436899404f23-util\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:07:22.805409 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.805382 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" Jul 15 19:07:22.805586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.805411 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np" event={"ID":"5e57dcbb-ff75-49cd-b516-260e548ea9ab","Type":"ContainerDied","Data":"f72c54777e46e7745a9595ffe03ee4ccafc1fbe6011d735f85e526ccb797cb71"} Jul 15 19:07:22.805586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.805460 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f72c54777e46e7745a9595ffe03ee4ccafc1fbe6011d735f85e526ccb797cb71" Jul 15 19:07:22.807357 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.807333 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" event={"ID":"433b012f-7bc3-4e6e-ac9f-436899404f23","Type":"ContainerDied","Data":"d1fedb1922707662c42cb0b9ba70f2ca80f715d0c6ea652b07007bba7e55ba96"} Jul 15 19:07:22.807357 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.807353 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp" Jul 15 19:07:22.807574 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:22.807390 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1fedb1922707662c42cb0b9ba70f2ca80f715d0c6ea652b07007bba7e55ba96" Jul 15 19:07:27.857340 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:27.857284 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:29.005039 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:29.005006 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:31.423096 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423061 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k"] Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423379 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423391 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423401 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423406 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423413 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423418 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423425 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423431 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423437 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423442 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423447 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423452 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423461 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423467 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423473 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423478 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerName="pull" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423484 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423490 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423495 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423499 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423508 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423516 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423527 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423532 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerName="util" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423574 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e57dcbb-ff75-49cd-b516-260e548ea9ab" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423583 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae4218b-87c0-488e-b2fc-1d3052adcc1d" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423590 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="433b012f-7bc3-4e6e-ac9f-436899404f23" containerName="extract" Jul 15 19:07:31.423583 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.423596 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71" containerName="extract" Jul 15 19:07:31.427882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.427867 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" Jul 15 19:07:31.430658 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.430633 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 15 19:07:31.430807 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.430794 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jtfdx\"" Jul 15 19:07:31.440928 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.440906 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k"] Jul 15 19:07:31.613948 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.613917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8gf\" (UniqueName: \"kubernetes.io/projected/b570ac71-b47e-4404-ba69-f567f7e7349e-kube-api-access-8n8gf\") pod \"dns-operator-controller-manager-864dd78c4c-vk66k\" (UID: \"b570ac71-b47e-4404-ba69-f567f7e7349e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" Jul 15 19:07:31.714789 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.714707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8gf\" (UniqueName: \"kubernetes.io/projected/b570ac71-b47e-4404-ba69-f567f7e7349e-kube-api-access-8n8gf\") pod \"dns-operator-controller-manager-864dd78c4c-vk66k\" (UID: \"b570ac71-b47e-4404-ba69-f567f7e7349e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" Jul 15 19:07:31.730941 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.730919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8gf\" (UniqueName: \"kubernetes.io/projected/b570ac71-b47e-4404-ba69-f567f7e7349e-kube-api-access-8n8gf\") pod \"dns-operator-controller-manager-864dd78c4c-vk66k\" (UID: \"b570ac71-b47e-4404-ba69-f567f7e7349e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" Jul 15 19:07:31.737754 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.737731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" Jul 15 19:07:31.882858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:31.882832 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k"] Jul 15 19:07:31.884797 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:31.884761 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb570ac71_b47e_4404_ba69_f567f7e7349e.slice/crio-5a8e317cfc31cefb00af00f9c86fb6f5885cd1622e9af507a95128b0a20208cb WatchSource:0}: Error finding container 5a8e317cfc31cefb00af00f9c86fb6f5885cd1622e9af507a95128b0a20208cb: Status 404 returned error can't find the container with id 5a8e317cfc31cefb00af00f9c86fb6f5885cd1622e9af507a95128b0a20208cb Jul 15 19:07:32.853361 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:32.853330 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" event={"ID":"b570ac71-b47e-4404-ba69-f567f7e7349e","Type":"ContainerStarted","Data":"5a8e317cfc31cefb00af00f9c86fb6f5885cd1622e9af507a95128b0a20208cb"} Jul 15 19:07:33.033943 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:33.033890 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:34.864299 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:34.864262 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" event={"ID":"b570ac71-b47e-4404-ba69-f567f7e7349e","Type":"ContainerStarted","Data":"220a7019c216e1ef10aa8c2da34ab52e1fc25bc38cdc7eac0d7c9a6af465c4be"} Jul 15 19:07:34.864689 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:34.864398 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" Jul 15 19:07:34.939570 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:34.939510 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" podStartSLOduration=1.463908961 podStartE2EDuration="3.939482237s" totalImagesPullingTime="2.475573276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:07:34.910217791 +0000 UTC m=+407.305573762" watchObservedRunningTime="2026-07-15 19:07:34.939482237 +0000 UTC m=+407.334838208" Jul 15 19:07:34.941144 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:34.941125 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf"] Jul 15 19:07:34.944890 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:34.944869 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:07:34.948080 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:34.948059 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dxbcf\"" Jul 15 19:07:34.960620 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:34.960595 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf"] Jul 15 19:07:35.044863 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.044824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckwg\" (UniqueName: \"kubernetes.io/projected/632a4557-b098-4b05-8794-4addc3d25d3e-kube-api-access-9ckwg\") pod \"limitador-operator-controller-manager-859d7b78bd-p5twf\" (UID: \"632a4557-b098-4b05-8794-4addc3d25d3e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:07:35.145591 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.145500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckwg\" (UniqueName: \"kubernetes.io/projected/632a4557-b098-4b05-8794-4addc3d25d3e-kube-api-access-9ckwg\") pod \"limitador-operator-controller-manager-859d7b78bd-p5twf\" (UID: \"632a4557-b098-4b05-8794-4addc3d25d3e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:07:35.155848 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.155820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckwg\" (UniqueName: \"kubernetes.io/projected/632a4557-b098-4b05-8794-4addc3d25d3e-kube-api-access-9ckwg\") pod \"limitador-operator-controller-manager-859d7b78bd-p5twf\" (UID: \"632a4557-b098-4b05-8794-4addc3d25d3e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:07:35.255544 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.255511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:07:35.398707 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.398680 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf"] Jul 15 19:07:35.400326 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:35.400299 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632a4557_b098_4b05_8794_4addc3d25d3e.slice/crio-33409f8b9e57c07c46e70a77fbe05d2ca462886215c7e208b61fc5756133ee97 WatchSource:0}: Error finding container 33409f8b9e57c07c46e70a77fbe05d2ca462886215c7e208b61fc5756133ee97: Status 404 returned error can't find the container with id 33409f8b9e57c07c46e70a77fbe05d2ca462886215c7e208b61fc5756133ee97 Jul 15 19:07:35.411585 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.411559 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:35.637433 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.637405 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:35.870156 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:35.870125 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" event={"ID":"632a4557-b098-4b05-8794-4addc3d25d3e","Type":"ContainerStarted","Data":"33409f8b9e57c07c46e70a77fbe05d2ca462886215c7e208b61fc5756133ee97"} Jul 15 19:07:36.328391 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:36.328359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:37.881433 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:37.881391 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" event={"ID":"632a4557-b098-4b05-8794-4addc3d25d3e","Type":"ContainerStarted","Data":"a0cd6f8f52332f7cb38ff55209bb012c53f8cef65b2aaed0fbbd031f2c96276d"} Jul 15 19:07:37.881952 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:37.881496 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:07:37.927933 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:37.927879 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" podStartSLOduration=2.176721095 podStartE2EDuration="3.927864767s" totalImagesPullingTime="1.751143672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:07:37.926282238 +0000 UTC m=+410.321638208" watchObservedRunningTime="2026-07-15 19:07:37.927864767 +0000 UTC m=+410.323220740" Jul 15 19:07:38.114372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:38.114332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:38.210971 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:38.210888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:38.888695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:38.888667 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt"] Jul 15 19:07:38.892009 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:38.891988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" Jul 15 19:07:38.896886 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:38.896864 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-njfcm\"" Jul 15 19:07:38.922187 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:38.922160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt"] Jul 15 19:07:38.975207 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:38.975181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpsn\" (UniqueName: \"kubernetes.io/projected/4f0e0590-6b30-4b59-9c86-adb58c91719b-kube-api-access-wlpsn\") pod \"authorino-operator-75cc8d5cb7-7vqqt\" (UID: \"4f0e0590-6b30-4b59-9c86-adb58c91719b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" Jul 15 19:07:39.075733 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:39.075702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpsn\" (UniqueName: \"kubernetes.io/projected/4f0e0590-6b30-4b59-9c86-adb58c91719b-kube-api-access-wlpsn\") pod \"authorino-operator-75cc8d5cb7-7vqqt\" (UID: \"4f0e0590-6b30-4b59-9c86-adb58c91719b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" Jul 15 19:07:39.088141 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:39.088106 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpsn\" (UniqueName: \"kubernetes.io/projected/4f0e0590-6b30-4b59-9c86-adb58c91719b-kube-api-access-wlpsn\") pod \"authorino-operator-75cc8d5cb7-7vqqt\" (UID: \"4f0e0590-6b30-4b59-9c86-adb58c91719b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" Jul 15 19:07:39.201811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:39.201724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" Jul 15 19:07:39.371822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:39.371798 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt"] Jul 15 19:07:39.373208 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:39.373183 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0e0590_6b30_4b59_9c86_adb58c91719b.slice/crio-e629385d2e354291204fdd23e856c8deb7412e62d093cf4fcb2e8b077c9db6b1 WatchSource:0}: Error finding container e629385d2e354291204fdd23e856c8deb7412e62d093cf4fcb2e8b077c9db6b1: Status 404 returned error can't find the container with id e629385d2e354291204fdd23e856c8deb7412e62d093cf4fcb2e8b077c9db6b1 Jul 15 19:07:39.628399 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:39.628369 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:39.892732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:39.892644 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" event={"ID":"4f0e0590-6b30-4b59-9c86-adb58c91719b","Type":"ContainerStarted","Data":"e629385d2e354291204fdd23e856c8deb7412e62d093cf4fcb2e8b077c9db6b1"} Jul 15 19:07:40.290433 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:40.290390 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:41.905360 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:41.905329 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" event={"ID":"4f0e0590-6b30-4b59-9c86-adb58c91719b","Type":"ContainerStarted","Data":"9a2655e8257db0fb484f02a912fdaea3acdd713b282e0a2ff732fb3e70a9a709"} Jul 15 19:07:41.905693 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:41.905453 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" Jul 15 19:07:41.927291 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:41.927235 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" podStartSLOduration=1.488173322 podStartE2EDuration="3.927219303s" totalImagesPullingTime="2.439045981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:07:41.926020421 +0000 UTC m=+414.321376406" watchObservedRunningTime="2026-07-15 19:07:41.927219303 +0000 UTC m=+414.322575273" Jul 15 19:07:42.272989 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:42.272942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:44.056454 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.056424 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44"] Jul 15 19:07:44.059882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.059863 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.063044 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.063027 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-klgdv\"" Jul 15 19:07:44.069122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.069098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:44.073167 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.073147 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44"] Jul 15 19:07:44.119641 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.119615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.119641 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.119646 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhxq\" (UniqueName: \"kubernetes.io/projected/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-kube-api-access-wmhxq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.220332 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.220298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.220484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.220335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhxq\" (UniqueName: \"kubernetes.io/projected/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-kube-api-access-wmhxq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.220681 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.220662 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.229576 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.229552 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhxq\" (UniqueName: \"kubernetes.io/projected/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-kube-api-access-wmhxq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.370042 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.369962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:44.526823 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.526799 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44"] Jul 15 19:07:44.527641 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:07:44.527616 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad82d07b_d64d_4590_a4fa_a5f3c64dfbbe.slice/crio-274910f47d05874906554c6fe1c3eebcff22a6cf6c6e7ada0e0d0ae183465fda WatchSource:0}: Error finding container 274910f47d05874906554c6fe1c3eebcff22a6cf6c6e7ada0e0d0ae183465fda: Status 404 returned error can't find the container with id 274910f47d05874906554c6fe1c3eebcff22a6cf6c6e7ada0e0d0ae183465fda Jul 15 19:07:44.920884 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:44.920853 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" event={"ID":"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe","Type":"ContainerStarted","Data":"274910f47d05874906554c6fe1c3eebcff22a6cf6c6e7ada0e0d0ae183465fda"} Jul 15 19:07:45.860543 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:45.860497 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:45.872193 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:45.872172 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vk66k" Jul 15 19:07:45.982021 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:45.981989 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:48.888685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:48.888653 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:07:49.448120 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:49.448084 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:49.560052 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:49.560016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:51.960537 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:51.960503 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" event={"ID":"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe","Type":"ContainerStarted","Data":"a1d1bfbb7ce5bfa0434b9f713710f72dcee9eecdcd1639708cb8cd7f5014bbec"} Jul 15 19:07:51.960993 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:51.960728 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:07:52.006824 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:52.006768 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" podStartSLOduration=1.51489213 podStartE2EDuration="8.006751622s" totalImagesPullingTime="6.491859492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:07:52.004447333 +0000 UTC m=+424.399803303" watchObservedRunningTime="2026-07-15 19:07:52.006751622 +0000 UTC m=+424.402107591" Jul 15 19:07:52.911703 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:52.911671 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7vqqt" Jul 15 19:07:53.792445 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:53.792349 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:53.871517 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:53.871483 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:53.952401 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:53.952367 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:54.177108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:54.177011 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:54.268896 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:54.268863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:57.206377 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:57.206339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:57.306689 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:57.306649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:57.950149 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:57.950110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:07:58.049487 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:07:58.049452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:02.968415 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:02.968388 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:04.627607 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.627570 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:04.750591 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.750555 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44"] Jul 15 19:08:04.750780 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.750758 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" containerName="manager" containerID="cri-o://a1d1bfbb7ce5bfa0434b9f713710f72dcee9eecdcd1639708cb8cd7f5014bbec" gracePeriod=2 Jul 15 19:08:04.757234 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.757204 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:04.759409 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.759386 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44"] Jul 15 19:08:04.839035 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.839002 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf"] Jul 15 19:08:04.839311 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.839259 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" containerName="manager" containerID="cri-o://a0cd6f8f52332f7cb38ff55209bb012c53f8cef65b2aaed0fbbd031f2c96276d" gracePeriod=2 Jul 15 19:08:04.841800 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:04.841768 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:04.842205 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.842182 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25"] Jul 15 19:08:04.842770 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.842745 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" containerName="manager" Jul 15 19:08:04.842859 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.842773 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" containerName="manager" Jul 15 19:08:04.842895 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.842861 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" containerName="manager" Jul 15 19:08:04.846068 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.846050 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:04.892925 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.892826 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf"] Jul 15 19:08:04.895708 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.895682 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25"] Jul 15 19:08:04.905934 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:04.905905 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:04.936966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.936937 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz"] Jul 15 19:08:04.937293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.937279 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" containerName="manager" Jul 15 19:08:04.937349 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.937294 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" containerName="manager" Jul 15 19:08:04.937384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.937360 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" containerName="manager" Jul 15 19:08:04.940361 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.940346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" Jul 15 19:08:04.961629 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:04.961464 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:04.969473 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.969449 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz"] Jul 15 19:08:04.986669 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.986635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tks25\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:04.986825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:04.986754 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7xz\" (UniqueName: \"kubernetes.io/projected/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-kube-api-access-ds7xz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tks25\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:05.004279 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:05.004221 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5twf\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:05.026099 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.026071 2582 generic.go:356] "Generic (PLEG): container finished" podID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" containerID="a1d1bfbb7ce5bfa0434b9f713710f72dcee9eecdcd1639708cb8cd7f5014bbec" exitCode=0 Jul 15 19:08:05.027611 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.027529 2582 generic.go:356] "Generic (PLEG): container finished" podID="632a4557-b098-4b05-8794-4addc3d25d3e" containerID="a0cd6f8f52332f7cb38ff55209bb012c53f8cef65b2aaed0fbbd031f2c96276d" exitCode=0 Jul 15 19:08:05.087397 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.087368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tks25\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:05.087507 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.087472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7xz\" (UniqueName: \"kubernetes.io/projected/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-kube-api-access-ds7xz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tks25\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:05.087574 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.087517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6f7\" (UniqueName: \"kubernetes.io/projected/b0833af5-0a67-496f-91cb-8b37d10f9c7f-kube-api-access-cr6f7\") pod \"limitador-operator-controller-manager-859d7b78bd-q94cz\" (UID: \"b0833af5-0a67-496f-91cb-8b37d10f9c7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" Jul 15 19:08:05.087775 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.087754 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tks25\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:05.092405 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.092388 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:05.095289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.095266 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:05.098736 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:05.098709 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:05.101280 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:05.101260 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5twf\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:05.103109 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.103090 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7xz\" (UniqueName: \"kubernetes.io/projected/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-kube-api-access-ds7xz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tks25\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:05.103598 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:05.103578 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:05.105211 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:05.105193 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5twf\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:05.188865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.188804 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmhxq\" (UniqueName: \"kubernetes.io/projected/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-kube-api-access-wmhxq\") pod \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " Jul 15 19:08:05.188865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.188846 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-extensions-socket-volume\") pod \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\" (UID: \"ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe\") " Jul 15 19:08:05.188997 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.188898 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ckwg\" (UniqueName: \"kubernetes.io/projected/632a4557-b098-4b05-8794-4addc3d25d3e-kube-api-access-9ckwg\") pod \"632a4557-b098-4b05-8794-4addc3d25d3e\" (UID: \"632a4557-b098-4b05-8794-4addc3d25d3e\") " Jul 15 19:08:05.189045 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.189026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6f7\" (UniqueName: \"kubernetes.io/projected/b0833af5-0a67-496f-91cb-8b37d10f9c7f-kube-api-access-cr6f7\") pod \"limitador-operator-controller-manager-859d7b78bd-q94cz\" (UID: \"b0833af5-0a67-496f-91cb-8b37d10f9c7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" Jul 15 19:08:05.189481 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.189453 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" (UID: "ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:08:05.190847 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.190823 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-kube-api-access-wmhxq" (OuterVolumeSpecName: "kube-api-access-wmhxq") pod "ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" (UID: "ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe"). InnerVolumeSpecName "kube-api-access-wmhxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:08:05.190938 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.190919 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632a4557-b098-4b05-8794-4addc3d25d3e-kube-api-access-9ckwg" (OuterVolumeSpecName: "kube-api-access-9ckwg") pod "632a4557-b098-4b05-8794-4addc3d25d3e" (UID: "632a4557-b098-4b05-8794-4addc3d25d3e"). InnerVolumeSpecName "kube-api-access-9ckwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:08:05.202474 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.202456 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6f7\" (UniqueName: \"kubernetes.io/projected/b0833af5-0a67-496f-91cb-8b37d10f9c7f-kube-api-access-cr6f7\") pod \"limitador-operator-controller-manager-859d7b78bd-q94cz\" (UID: \"b0833af5-0a67-496f-91cb-8b37d10f9c7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" Jul 15 19:08:05.229306 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.229283 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:05.252358 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.252335 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" Jul 15 19:08:05.291152 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.290702 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-extensions-socket-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:05.291152 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.290735 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckwg\" (UniqueName: \"kubernetes.io/projected/632a4557-b098-4b05-8794-4addc3d25d3e-kube-api-access-9ckwg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:05.291152 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.290749 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhxq\" (UniqueName: \"kubernetes.io/projected/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe-kube-api-access-wmhxq\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:05.372633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.372611 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25"] Jul 15 19:08:05.374714 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:08:05.374682 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93fa1b9_5a96_4ddb_b74e_a07ebed43d67.slice/crio-a26ab8a42680acc2f330cc2cf3d9235737521da973f40900e8864596a0a1db66 WatchSource:0}: Error finding container a26ab8a42680acc2f330cc2cf3d9235737521da973f40900e8864596a0a1db66: Status 404 returned error can't find the container with id a26ab8a42680acc2f330cc2cf3d9235737521da973f40900e8864596a0a1db66 Jul 15 19:08:05.390265 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:05.390197 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz"] Jul 15 19:08:05.392285 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:08:05.392259 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0833af5_0a67_496f_91cb_8b37d10f9c7f.slice/crio-9ca2112b466abf72d617a16f74850dd01df7a7b2164369528bad124ca7b26ba7 WatchSource:0}: Error finding container 9ca2112b466abf72d617a16f74850dd01df7a7b2164369528bad124ca7b26ba7: Status 404 returned error can't find the container with id 9ca2112b466abf72d617a16f74850dd01df7a7b2164369528bad124ca7b26ba7 Jul 15 19:08:06.033296 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.033209 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:06.033296 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.033219 2582 scope.go:117] "RemoveContainer" containerID="a1d1bfbb7ce5bfa0434b9f713710f72dcee9eecdcd1639708cb8cd7f5014bbec" Jul 15 19:08:06.034964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.034934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" event={"ID":"b0833af5-0a67-496f-91cb-8b37d10f9c7f","Type":"ContainerStarted","Data":"924e3b0133c25b767f10d55dc207a1cd5e949604b3c489618f3b98282316aef3"} Jul 15 19:08:06.035077 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.034988 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" event={"ID":"b0833af5-0a67-496f-91cb-8b37d10f9c7f","Type":"ContainerStarted","Data":"9ca2112b466abf72d617a16f74850dd01df7a7b2164369528bad124ca7b26ba7"} Jul 15 19:08:06.035077 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.035052 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" Jul 15 19:08:06.035688 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:06.035659 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:06.036401 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.036380 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:06.037724 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:06.037698 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5twf\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:06.038141 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.038122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" event={"ID":"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67","Type":"ContainerStarted","Data":"f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22"} Jul 15 19:08:06.038253 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.038161 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" event={"ID":"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67","Type":"ContainerStarted","Data":"a26ab8a42680acc2f330cc2cf3d9235737521da973f40900e8864596a0a1db66"} Jul 15 19:08:06.038253 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.038207 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:06.039713 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:06.039681 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:06.044425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.044406 2582 scope.go:117] "RemoveContainer" containerID="a0cd6f8f52332f7cb38ff55209bb012c53f8cef65b2aaed0fbbd031f2c96276d" Jul 15 19:08:06.058216 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:06.058193 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5twf\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:06.058751 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.058716 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" podStartSLOduration=2.058704471 podStartE2EDuration="2.058704471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:06.055745491 +0000 UTC m=+438.451101460" watchObservedRunningTime="2026-07-15 19:08:06.058704471 +0000 UTC m=+438.454060422" Jul 15 19:08:06.060092 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:06.060073 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5twf\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5twf" Jul 15 19:08:06.061601 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:06.061582 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rdj44\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdj44" Jul 15 19:08:06.087681 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.087632 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" podStartSLOduration=2.087618151 podStartE2EDuration="2.087618151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:06.085362258 +0000 UTC m=+438.480718239" watchObservedRunningTime="2026-07-15 19:08:06.087618151 +0000 UTC m=+438.482974144" Jul 15 19:08:06.198994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.198963 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="632a4557-b098-4b05-8794-4addc3d25d3e" path="/var/lib/kubelet/pods/632a4557-b098-4b05-8794-4addc3d25d3e/volumes" Jul 15 19:08:06.199328 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:06.199315 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe" path="/var/lib/kubelet/pods/ad82d07b-d64d-4590-a4fa-a5f3c64dfbbe/volumes" Jul 15 19:08:07.819617 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:07.819584 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:07.918679 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:07.918649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:12.242690 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:12.242600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:12.343370 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:12.343333 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:12.445364 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:12.445328 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:16.202215 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:16.202170 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:16.310433 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:16.310389 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:17.048404 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:17.048376 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:17.048584 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:17.048457 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q94cz" Jul 15 19:08:23.256232 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.256193 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25"] Jul 15 19:08:23.256656 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.256419 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" podUID="f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" containerName="manager" containerID="cri-o://f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22" gracePeriod=10 Jul 15 19:08:23.545960 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.545934 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:23.649481 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.649445 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-extensions-socket-volume\") pod \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " Jul 15 19:08:23.649671 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.649524 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds7xz\" (UniqueName: \"kubernetes.io/projected/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-kube-api-access-ds7xz\") pod \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\" (UID: \"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67\") " Jul 15 19:08:23.652050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.650204 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" (UID: "f93fa1b9-5a96-4ddb-b74e-a07ebed43d67"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:08:23.652633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.652529 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-kube-api-access-ds7xz" (OuterVolumeSpecName: "kube-api-access-ds7xz") pod "f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" (UID: "f93fa1b9-5a96-4ddb-b74e-a07ebed43d67"). InnerVolumeSpecName "kube-api-access-ds7xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:08:23.750627 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.750589 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-extensions-socket-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:23.750627 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:23.750621 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds7xz\" (UniqueName: \"kubernetes.io/projected/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67-kube-api-access-ds7xz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:24.128441 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.128402 2582 generic.go:356] "Generic (PLEG): container finished" podID="f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" containerID="f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22" exitCode=0 Jul 15 19:08:24.128611 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.128443 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" event={"ID":"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67","Type":"ContainerDied","Data":"f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22"} Jul 15 19:08:24.128611 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.128480 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" Jul 15 19:08:24.128611 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.128529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25" event={"ID":"f93fa1b9-5a96-4ddb-b74e-a07ebed43d67","Type":"ContainerDied","Data":"a26ab8a42680acc2f330cc2cf3d9235737521da973f40900e8864596a0a1db66"} Jul 15 19:08:24.128611 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.128554 2582 scope.go:117] "RemoveContainer" containerID="f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22" Jul 15 19:08:24.137444 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.137423 2582 scope.go:117] "RemoveContainer" containerID="f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22" Jul 15 19:08:24.137697 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:24.137677 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22\": container with ID starting with f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22 not found: ID does not exist" containerID="f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22" Jul 15 19:08:24.137760 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.137727 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22"} err="failed to get container status \"f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22\": rpc error: code = NotFound desc = could not find container \"f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22\": container with ID starting with f91b2beb4e6d13b5bb6e376a1ae624cae87d02bd9bae5be362c072350130cd22 not found: ID does not exist" Jul 15 19:08:24.151868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.151844 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25"] Jul 15 19:08:24.156124 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.156102 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tks25"] Jul 15 19:08:24.200049 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:24.200025 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" path="/var/lib/kubelet/pods/f93fa1b9-5a96-4ddb-b74e-a07ebed43d67/volumes" Jul 15 19:08:29.861001 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:29.860963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:39.774218 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.774190 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb"] Jul 15 19:08:39.774652 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.774635 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" containerName="manager" Jul 15 19:08:39.774706 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.774654 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" containerName="manager" Jul 15 19:08:39.774742 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.774714 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f93fa1b9-5a96-4ddb-b74e-a07ebed43d67" containerName="manager" Jul 15 19:08:39.777441 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.777421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.784345 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.784315 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z28vr\"" Jul 15 19:08:39.798852 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.798823 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb"] Jul 15 19:08:39.817536 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.817505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:39.866164 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scz6p\" (UniqueName: \"kubernetes.io/projected/e67fea28-4b13-4985-9a30-b34c5460e8c8-kube-api-access-scz6p\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866470 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fea28-4b13-4985-9a30-b34c5460e8c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866470 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866470 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866642 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.866642 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.866517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.966904 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.966873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.966904 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.966908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scz6p\" (UniqueName: \"kubernetes.io/projected/e67fea28-4b13-4985-9a30-b34c5460e8c8-kube-api-access-scz6p\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.966936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.966966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fea28-4b13-4985-9a30-b34c5460e8c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.966987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.967002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.967038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.967057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.967104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967755 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.967729 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967839 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.967780 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.967950 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.967837 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.968372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.968349 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.968568 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.968546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fea28-4b13-4985-9a30-b34c5460e8c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.974347 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.974322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.974690 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.974672 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.994012 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.993984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fea28-4b13-4985-9a30-b34c5460e8c8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:39.995832 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:39.995811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scz6p\" (UniqueName: \"kubernetes.io/projected/e67fea28-4b13-4985-9a30-b34c5460e8c8-kube-api-access-scz6p\") pod \"maas-default-gateway-openshift-default-56654dcf7-79zpb\" (UID: \"e67fea28-4b13-4985-9a30-b34c5460e8c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:40.085466 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:40.085364 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:40.089184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:40.089163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:40.222294 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:40.222267 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb"] Jul 15 19:08:40.224059 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:08:40.224032 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67fea28_4b13_4985_9a30_b34c5460e8c8.slice/crio-8d235aec2551791f8d56ace072564d10fb935cd0740a08a612074d202ed9428f WatchSource:0}: Error finding container 8d235aec2551791f8d56ace072564d10fb935cd0740a08a612074d202ed9428f: Status 404 returned error can't find the container with id 8d235aec2551791f8d56ace072564d10fb935cd0740a08a612074d202ed9428f Jul 15 19:08:40.225904 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:40.225872 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:08:40.226007 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:40.225947 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:08:40.226007 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:40.225992 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:08:41.210413 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:41.210376 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" event={"ID":"e67fea28-4b13-4985-9a30-b34c5460e8c8","Type":"ContainerStarted","Data":"4bbdafdd83cc406e0e4dd929a6ee23f44aca3a0bc6381b813e7bd43125277c87"} Jul 15 19:08:41.210784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:41.210432 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" event={"ID":"e67fea28-4b13-4985-9a30-b34c5460e8c8","Type":"ContainerStarted","Data":"8d235aec2551791f8d56ace072564d10fb935cd0740a08a612074d202ed9428f"} Jul 15 19:08:41.241713 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:41.241667 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" podStartSLOduration=2.241652322 podStartE2EDuration="2.241652322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:41.235005339 +0000 UTC m=+473.630361321" watchObservedRunningTime="2026-07-15 19:08:41.241652322 +0000 UTC m=+473.637008292" Jul 15 19:08:42.090195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.090164 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:42.094772 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.094747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:42.215023 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.214998 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:42.215815 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.215796 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:08:42.395979 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.395904 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcctn"] Jul 15 19:08:42.399135 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.399119 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.401709 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.401686 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 19:08:42.401827 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.401724 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n6qmk\"" Jul 15 19:08:42.403366 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.403331 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:42.410694 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.410666 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcctn"] Jul 15 19:08:42.483609 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.483580 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcctn"] Jul 15 19:08:42.486637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.486608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttv8s\" (UniqueName: \"kubernetes.io/projected/eed2470e-b284-46e0-a00f-fadf34bf499c-kube-api-access-ttv8s\") pod \"limitador-limitador-64c94fd64f-kcctn\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.486800 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.486654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed2470e-b284-46e0-a00f-fadf34bf499c-config-file\") pod \"limitador-limitador-64c94fd64f-kcctn\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.492229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.492199 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:42.587803 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.587769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttv8s\" (UniqueName: \"kubernetes.io/projected/eed2470e-b284-46e0-a00f-fadf34bf499c-kube-api-access-ttv8s\") pod \"limitador-limitador-64c94fd64f-kcctn\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.588009 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.587828 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed2470e-b284-46e0-a00f-fadf34bf499c-config-file\") pod \"limitador-limitador-64c94fd64f-kcctn\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.588551 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.588524 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed2470e-b284-46e0-a00f-fadf34bf499c-config-file\") pod \"limitador-limitador-64c94fd64f-kcctn\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.592266 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.592203 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:42.601139 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.601114 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttv8s\" (UniqueName: \"kubernetes.io/projected/eed2470e-b284-46e0-a00f-fadf34bf499c-kube-api-access-ttv8s\") pod \"limitador-limitador-64c94fd64f-kcctn\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.711181 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.711103 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:42.841143 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:42.841117 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcctn"] Jul 15 19:08:42.842957 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:08:42.842930 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed2470e_b284_46e0_a00f_fadf34bf499c.slice/crio-5355ea2c48aa34b983f7a879b09152405733b1e1576fd34dc20c94fdd8db5e53 WatchSource:0}: Error finding container 5355ea2c48aa34b983f7a879b09152405733b1e1576fd34dc20c94fdd8db5e53: Status 404 returned error can't find the container with id 5355ea2c48aa34b983f7a879b09152405733b1e1576fd34dc20c94fdd8db5e53 Jul 15 19:08:43.205859 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.205825 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-bwlq9"] Jul 15 19:08:43.210270 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.210231 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bwlq9" Jul 15 19:08:43.212688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.212672 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ll64q\"" Jul 15 19:08:43.217949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.217863 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bwlq9"] Jul 15 19:08:43.221213 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.221192 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" event={"ID":"eed2470e-b284-46e0-a00f-fadf34bf499c","Type":"ContainerStarted","Data":"5355ea2c48aa34b983f7a879b09152405733b1e1576fd34dc20c94fdd8db5e53"} Jul 15 19:08:43.247998 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.247971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:43.295194 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.295160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tvl\" (UniqueName: \"kubernetes.io/projected/4978d045-a660-478f-b6b1-76a50d4ddf86-kube-api-access-66tvl\") pod \"authorino-58cd955675-bwlq9\" (UID: \"4978d045-a660-478f-b6b1-76a50d4ddf86\") " pod="kuadrant-system/authorino-58cd955675-bwlq9" Jul 15 19:08:43.326816 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.326789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:43.396176 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.396141 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66tvl\" (UniqueName: \"kubernetes.io/projected/4978d045-a660-478f-b6b1-76a50d4ddf86-kube-api-access-66tvl\") pod \"authorino-58cd955675-bwlq9\" (UID: \"4978d045-a660-478f-b6b1-76a50d4ddf86\") " pod="kuadrant-system/authorino-58cd955675-bwlq9" Jul 15 19:08:43.406130 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.406107 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tvl\" (UniqueName: \"kubernetes.io/projected/4978d045-a660-478f-b6b1-76a50d4ddf86-kube-api-access-66tvl\") pod \"authorino-58cd955675-bwlq9\" (UID: \"4978d045-a660-478f-b6b1-76a50d4ddf86\") " pod="kuadrant-system/authorino-58cd955675-bwlq9" Jul 15 19:08:43.466392 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.466306 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-wmqzj"] Jul 15 19:08:43.470047 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.470020 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wmqzj"] Jul 15 19:08:43.470158 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.470064 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wmqzj" Jul 15 19:08:43.496557 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.496534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz444\" (UniqueName: \"kubernetes.io/projected/ee4af733-9e54-4115-9b69-9f1d7cb064ac-kube-api-access-lz444\") pod \"authorino-75d556bd77-wmqzj\" (UID: \"ee4af733-9e54-4115-9b69-9f1d7cb064ac\") " pod="kuadrant-system/authorino-75d556bd77-wmqzj" Jul 15 19:08:43.521473 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.521436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bwlq9" Jul 15 19:08:43.597795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.597761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz444\" (UniqueName: \"kubernetes.io/projected/ee4af733-9e54-4115-9b69-9f1d7cb064ac-kube-api-access-lz444\") pod \"authorino-75d556bd77-wmqzj\" (UID: \"ee4af733-9e54-4115-9b69-9f1d7cb064ac\") " pod="kuadrant-system/authorino-75d556bd77-wmqzj" Jul 15 19:08:43.607391 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.607369 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz444\" (UniqueName: \"kubernetes.io/projected/ee4af733-9e54-4115-9b69-9f1d7cb064ac-kube-api-access-lz444\") pod \"authorino-75d556bd77-wmqzj\" (UID: \"ee4af733-9e54-4115-9b69-9f1d7cb064ac\") " pod="kuadrant-system/authorino-75d556bd77-wmqzj" Jul 15 19:08:43.652577 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.652556 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bwlq9"] Jul 15 19:08:43.654727 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:08:43.654695 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4978d045_a660_478f_b6b1_76a50d4ddf86.slice/crio-ffecab72c267ad252d39763eb1f1aecf73d68d861cf74433fb4d21e3e694907a WatchSource:0}: Error finding container ffecab72c267ad252d39763eb1f1aecf73d68d861cf74433fb4d21e3e694907a: Status 404 returned error can't find the container with id ffecab72c267ad252d39763eb1f1aecf73d68d861cf74433fb4d21e3e694907a Jul 15 19:08:43.780145 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.780113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wmqzj" Jul 15 19:08:43.940469 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:43.940410 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wmqzj"] Jul 15 19:08:43.942658 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:08:43.942628 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee4af733_9e54_4115_9b69_9f1d7cb064ac.slice/crio-1d9f278dd179b907d22eca935d8b4c1273c1b8be7eebf73379fa7570279211c5 WatchSource:0}: Error finding container 1d9f278dd179b907d22eca935d8b4c1273c1b8be7eebf73379fa7570279211c5: Status 404 returned error can't find the container with id 1d9f278dd179b907d22eca935d8b4c1273c1b8be7eebf73379fa7570279211c5 Jul 15 19:08:44.228171 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:44.228136 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bwlq9" event={"ID":"4978d045-a660-478f-b6b1-76a50d4ddf86","Type":"ContainerStarted","Data":"ffecab72c267ad252d39763eb1f1aecf73d68d861cf74433fb4d21e3e694907a"} Jul 15 19:08:44.229543 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:44.229514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wmqzj" event={"ID":"ee4af733-9e54-4115-9b69-9f1d7cb064ac","Type":"ContainerStarted","Data":"1d9f278dd179b907d22eca935d8b4c1273c1b8be7eebf73379fa7570279211c5"} Jul 15 19:08:49.268718 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.268678 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wmqzj" event={"ID":"ee4af733-9e54-4115-9b69-9f1d7cb064ac","Type":"ContainerStarted","Data":"dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459"} Jul 15 19:08:49.270082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.270049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" event={"ID":"eed2470e-b284-46e0-a00f-fadf34bf499c","Type":"ContainerStarted","Data":"51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c"} Jul 15 19:08:49.270205 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.270140 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:49.271330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.271309 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bwlq9" event={"ID":"4978d045-a660-478f-b6b1-76a50d4ddf86","Type":"ContainerStarted","Data":"4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865"} Jul 15 19:08:49.286117 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.286070 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-wmqzj" podStartSLOduration=2.046850565 podStartE2EDuration="6.286058441s" totalImagesPullingTime="4.239207876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:49.284153037 +0000 UTC m=+481.679509007" watchObservedRunningTime="2026-07-15 19:08:49.286058441 +0000 UTC m=+481.681414411" Jul 15 19:08:49.302049 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.302002 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" podStartSLOduration=1.953436315 podStartE2EDuration="7.301990558s" totalImagesPullingTime="5.348554243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:49.300113155 +0000 UTC m=+481.695469125" watchObservedRunningTime="2026-07-15 19:08:49.301990558 +0000 UTC m=+481.697346528" Jul 15 19:08:49.310629 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.310599 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bwlq9"] Jul 15 19:08:49.326684 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:49.326641 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-bwlq9" podStartSLOduration=1.8339007729999999 podStartE2EDuration="6.326627974s" totalImagesPullingTime="4.492727201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:49.324235107 +0000 UTC m=+481.719591055" watchObservedRunningTime="2026-07-15 19:08:49.326627974 +0000 UTC m=+481.721983943" Jul 15 19:08:51.280447 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:51.280410 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-bwlq9" podUID="4978d045-a660-478f-b6b1-76a50d4ddf86" containerName="authorino" containerID="cri-o://4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865" gracePeriod=30 Jul 15 19:08:51.527048 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:51.527026 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bwlq9" Jul 15 19:08:51.577169 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:51.577105 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66tvl\" (UniqueName: \"kubernetes.io/projected/4978d045-a660-478f-b6b1-76a50d4ddf86-kube-api-access-66tvl\") pod \"4978d045-a660-478f-b6b1-76a50d4ddf86\" (UID: \"4978d045-a660-478f-b6b1-76a50d4ddf86\") " Jul 15 19:08:51.579113 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:51.579085 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4978d045-a660-478f-b6b1-76a50d4ddf86-kube-api-access-66tvl" (OuterVolumeSpecName: "kube-api-access-66tvl") pod "4978d045-a660-478f-b6b1-76a50d4ddf86" (UID: "4978d045-a660-478f-b6b1-76a50d4ddf86"). InnerVolumeSpecName "kube-api-access-66tvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:08:51.678652 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:51.678622 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66tvl\" (UniqueName: \"kubernetes.io/projected/4978d045-a660-478f-b6b1-76a50d4ddf86-kube-api-access-66tvl\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:52.286618 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.286583 2582 generic.go:356] "Generic (PLEG): container finished" podID="4978d045-a660-478f-b6b1-76a50d4ddf86" containerID="4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865" exitCode=0 Jul 15 19:08:52.286618 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.286622 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bwlq9" Jul 15 19:08:52.287085 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.286663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bwlq9" event={"ID":"4978d045-a660-478f-b6b1-76a50d4ddf86","Type":"ContainerDied","Data":"4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865"} Jul 15 19:08:52.287085 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.286704 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bwlq9" event={"ID":"4978d045-a660-478f-b6b1-76a50d4ddf86","Type":"ContainerDied","Data":"ffecab72c267ad252d39763eb1f1aecf73d68d861cf74433fb4d21e3e694907a"} Jul 15 19:08:52.287085 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.286721 2582 scope.go:117] "RemoveContainer" containerID="4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865" Jul 15 19:08:52.295184 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.295165 2582 scope.go:117] "RemoveContainer" containerID="4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865" Jul 15 19:08:52.295449 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:52.295432 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865\": container with ID starting with 4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865 not found: ID does not exist" containerID="4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865" Jul 15 19:08:52.295520 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.295459 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865"} err="failed to get container status \"4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865\": rpc error: code = NotFound desc = could not find container \"4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865\": container with ID starting with 4a5bd6c9bc486197188eaead9d089c580f275ee347dbab3369f3af869d493865 not found: ID does not exist" Jul 15 19:08:52.304070 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.304047 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bwlq9"] Jul 15 19:08:52.308634 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:52.308612 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-bwlq9"] Jul 15 19:08:54.198907 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:54.198870 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4978d045-a660-478f-b6b1-76a50d4ddf86" path="/var/lib/kubelet/pods/4978d045-a660-478f-b6b1-76a50d4ddf86/volumes" Jul 15 19:08:57.160653 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:57.160614 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:08:58.183865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:58.183823 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcctn"] Jul 15 19:08:58.184353 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:58.184063 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" podUID="eed2470e-b284-46e0-a00f-fadf34bf499c" containerName="limitador" containerID="cri-o://51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c" gracePeriod=30 Jul 15 19:08:58.186601 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:58.186577 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:59.135786 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.135761 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:59.241994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.241952 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed2470e-b284-46e0-a00f-fadf34bf499c-config-file\") pod \"eed2470e-b284-46e0-a00f-fadf34bf499c\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " Jul 15 19:08:59.242458 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.242019 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttv8s\" (UniqueName: \"kubernetes.io/projected/eed2470e-b284-46e0-a00f-fadf34bf499c-kube-api-access-ttv8s\") pod \"eed2470e-b284-46e0-a00f-fadf34bf499c\" (UID: \"eed2470e-b284-46e0-a00f-fadf34bf499c\") " Jul 15 19:08:59.242458 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.242411 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed2470e-b284-46e0-a00f-fadf34bf499c-config-file" (OuterVolumeSpecName: "config-file") pod "eed2470e-b284-46e0-a00f-fadf34bf499c" (UID: "eed2470e-b284-46e0-a00f-fadf34bf499c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:08:59.244047 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.244022 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed2470e-b284-46e0-a00f-fadf34bf499c-kube-api-access-ttv8s" (OuterVolumeSpecName: "kube-api-access-ttv8s") pod "eed2470e-b284-46e0-a00f-fadf34bf499c" (UID: "eed2470e-b284-46e0-a00f-fadf34bf499c"). InnerVolumeSpecName "kube-api-access-ttv8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:08:59.321814 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.321781 2582 generic.go:356] "Generic (PLEG): container finished" podID="eed2470e-b284-46e0-a00f-fadf34bf499c" containerID="51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c" exitCode=0 Jul 15 19:08:59.321991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.321829 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" Jul 15 19:08:59.321991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.321859 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" event={"ID":"eed2470e-b284-46e0-a00f-fadf34bf499c","Type":"ContainerDied","Data":"51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c"} Jul 15 19:08:59.321991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.321907 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcctn" event={"ID":"eed2470e-b284-46e0-a00f-fadf34bf499c","Type":"ContainerDied","Data":"5355ea2c48aa34b983f7a879b09152405733b1e1576fd34dc20c94fdd8db5e53"} Jul 15 19:08:59.321991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.321929 2582 scope.go:117] "RemoveContainer" containerID="51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c" Jul 15 19:08:59.330410 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.330395 2582 scope.go:117] "RemoveContainer" containerID="51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c" Jul 15 19:08:59.330654 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:08:59.330634 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c\": container with ID starting with 51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c not found: ID does not exist" containerID="51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c" Jul 15 19:08:59.330711 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.330665 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c"} err="failed to get container status \"51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c\": rpc error: code = NotFound desc = could not find container \"51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c\": container with ID starting with 51cc28da267788aac0569d011a397f8a9900b73614a555a5b9bb63b7832d388c not found: ID does not exist" Jul 15 19:08:59.342626 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.342607 2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed2470e-b284-46e0-a00f-fadf34bf499c-config-file\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:59.342626 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.342627 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttv8s\" (UniqueName: \"kubernetes.io/projected/eed2470e-b284-46e0-a00f-fadf34bf499c-kube-api-access-ttv8s\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:08:59.345294 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.345273 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcctn"] Jul 15 19:08:59.349011 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:08:59.348990 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcctn"] Jul 15 19:09:00.198464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:00.198435 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed2470e-b284-46e0-a00f-fadf34bf499c" path="/var/lib/kubelet/pods/eed2470e-b284-46e0-a00f-fadf34bf499c/volumes" Jul 15 19:09:03.652118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:03.652082 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:03.723791 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:03.723752 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:04.043100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.043064 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm"] Jul 15 19:09:04.043423 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.043410 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed2470e-b284-46e0-a00f-fadf34bf499c" containerName="limitador" Jul 15 19:09:04.043484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.043425 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed2470e-b284-46e0-a00f-fadf34bf499c" containerName="limitador" Jul 15 19:09:04.043484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.043435 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4978d045-a660-478f-b6b1-76a50d4ddf86" containerName="authorino" Jul 15 19:09:04.043484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.043443 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4978d045-a660-478f-b6b1-76a50d4ddf86" containerName="authorino" Jul 15 19:09:04.043596 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.043487 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed2470e-b284-46e0-a00f-fadf34bf499c" containerName="limitador" Jul 15 19:09:04.043596 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.043498 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4978d045-a660-478f-b6b1-76a50d4ddf86" containerName="authorino" Jul 15 19:09:04.049883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.048695 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.051777 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.051756 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 19:09:04.051881 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.051833 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8zwbp\"" Jul 15 19:09:04.052668 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.052639 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 19:09:04.052798 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.052671 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 15 19:09:04.060569 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.060547 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm"] Jul 15 19:09:04.081318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.081284 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:04.187682 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.187645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94qb\" (UniqueName: \"kubernetes.io/projected/1897f6e9-366f-48f8-b863-f4549f4f52ec-kube-api-access-d94qb\") pod \"postgres-69fb4594cf-zhsqm\" (UID: \"1897f6e9-366f-48f8-b863-f4549f4f52ec\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.187854 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.187723 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1897f6e9-366f-48f8-b863-f4549f4f52ec-data\") pod \"postgres-69fb4594cf-zhsqm\" (UID: \"1897f6e9-366f-48f8-b863-f4549f4f52ec\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.288289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.288234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1897f6e9-366f-48f8-b863-f4549f4f52ec-data\") pod \"postgres-69fb4594cf-zhsqm\" (UID: \"1897f6e9-366f-48f8-b863-f4549f4f52ec\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.288470 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.288332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94qb\" (UniqueName: \"kubernetes.io/projected/1897f6e9-366f-48f8-b863-f4549f4f52ec-kube-api-access-d94qb\") pod \"postgres-69fb4594cf-zhsqm\" (UID: \"1897f6e9-366f-48f8-b863-f4549f4f52ec\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.288613 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.288595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1897f6e9-366f-48f8-b863-f4549f4f52ec-data\") pod \"postgres-69fb4594cf-zhsqm\" (UID: \"1897f6e9-366f-48f8-b863-f4549f4f52ec\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.297399 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.297338 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94qb\" (UniqueName: \"kubernetes.io/projected/1897f6e9-366f-48f8-b863-f4549f4f52ec-kube-api-access-d94qb\") pod \"postgres-69fb4594cf-zhsqm\" (UID: \"1897f6e9-366f-48f8-b863-f4549f4f52ec\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.360809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.360771 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:04.503897 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:04.503870 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm"] Jul 15 19:09:04.505627 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:09:04.505601 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1897f6e9_366f_48f8_b863_f4549f4f52ec.slice/crio-9089f9b859c45054dbd6d2696b79ffe37c7d6b331d5dfe8a0a4070ceda45a3be WatchSource:0}: Error finding container 9089f9b859c45054dbd6d2696b79ffe37c7d6b331d5dfe8a0a4070ceda45a3be: Status 404 returned error can't find the container with id 9089f9b859c45054dbd6d2696b79ffe37c7d6b331d5dfe8a0a4070ceda45a3be Jul 15 19:09:05.359731 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:05.359684 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" event={"ID":"1897f6e9-366f-48f8-b863-f4549f4f52ec","Type":"ContainerStarted","Data":"9089f9b859c45054dbd6d2696b79ffe37c7d6b331d5dfe8a0a4070ceda45a3be"} Jul 15 19:09:09.381945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:09.381905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" event={"ID":"1897f6e9-366f-48f8-b863-f4549f4f52ec","Type":"ContainerStarted","Data":"a0a874e7d2915803f3ff85581077727cc97fa71bbd6a98dcbccc53b3e5d99235"} Jul 15 19:09:09.382344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:09.382023 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:09.399891 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:09.399842 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" podStartSLOduration=0.683816249 podStartE2EDuration="5.399826441s" totalImagesPullingTime="4.716010192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:09:09.397963875 +0000 UTC m=+501.793319839" watchObservedRunningTime="2026-07-15 19:09:09.399826441 +0000 UTC m=+501.795182411" Jul 15 19:09:15.416606 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:15.416578 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhsqm" Jul 15 19:09:15.857410 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:15.857372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:15.956008 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:15.955972 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:15.982485 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:15.982456 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rpfj4"] Jul 15 19:09:15.989007 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:15.988980 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rpfj4" Jul 15 19:09:15.991307 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:15.991282 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rpfj4"] Jul 15 19:09:16.092416 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.092377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sj5x\" (UniqueName: \"kubernetes.io/projected/96b2faaa-9c86-4109-a3b0-21ae6b4eb60d-kube-api-access-4sj5x\") pod \"authorino-5d4b67c446-rpfj4\" (UID: \"96b2faaa-9c86-4109-a3b0-21ae6b4eb60d\") " pod="kuadrant-system/authorino-5d4b67c446-rpfj4" Jul 15 19:09:16.178480 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.178403 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rpfj4"] Jul 15 19:09:16.178672 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:09:16.178655 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4sj5x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-rpfj4" podUID="96b2faaa-9c86-4109-a3b0-21ae6b4eb60d" Jul 15 19:09:16.193319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.193291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sj5x\" (UniqueName: \"kubernetes.io/projected/96b2faaa-9c86-4109-a3b0-21ae6b4eb60d-kube-api-access-4sj5x\") pod \"authorino-5d4b67c446-rpfj4\" (UID: \"96b2faaa-9c86-4109-a3b0-21ae6b4eb60d\") " pod="kuadrant-system/authorino-5d4b67c446-rpfj4" Jul 15 19:09:16.207546 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.207519 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-pnlc9"] Jul 15 19:09:16.210226 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.210205 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sj5x\" (UniqueName: \"kubernetes.io/projected/96b2faaa-9c86-4109-a3b0-21ae6b4eb60d-kube-api-access-4sj5x\") pod \"authorino-5d4b67c446-rpfj4\" (UID: \"96b2faaa-9c86-4109-a3b0-21ae6b4eb60d\") " pod="kuadrant-system/authorino-5d4b67c446-rpfj4" Jul 15 19:09:16.211151 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.211136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.213624 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.213607 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 19:09:16.220021 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.219998 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-pnlc9"] Jul 15 19:09:16.242540 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.242509 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-pnlc9"] Jul 15 19:09:16.242766 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:09:16.242748 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-47tfp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-pnlc9" podUID="2dae1383-0ea3-4af6-adeb-097ea975a250" Jul 15 19:09:16.275850 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.275816 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bcf6598c4-8z2ct"] Jul 15 19:09:16.279452 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.279434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.287121 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.287097 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcf6598c4-8z2ct"] Jul 15 19:09:16.294695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.294658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tfp\" (UniqueName: \"kubernetes.io/projected/2dae1383-0ea3-4af6-adeb-097ea975a250-kube-api-access-47tfp\") pod \"authorino-76877994-pnlc9\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.294840 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.294713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dae1383-0ea3-4af6-adeb-097ea975a250-tls-cert\") pod \"authorino-76877994-pnlc9\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.395853 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.395818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc99a172-de85-46d6-ad47-0b35bb9a846c-tls-cert\") pod \"authorino-bcf6598c4-8z2ct\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.396022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.395879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/fc99a172-de85-46d6-ad47-0b35bb9a846c-kube-api-access-57wpr\") pod \"authorino-bcf6598c4-8z2ct\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.396022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.395971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tfp\" (UniqueName: \"kubernetes.io/projected/2dae1383-0ea3-4af6-adeb-097ea975a250-kube-api-access-47tfp\") pod \"authorino-76877994-pnlc9\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.396118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.396027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dae1383-0ea3-4af6-adeb-097ea975a250-tls-cert\") pod \"authorino-76877994-pnlc9\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.398448 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.398424 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dae1383-0ea3-4af6-adeb-097ea975a250-tls-cert\") pod \"authorino-76877994-pnlc9\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.406921 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.406904 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tfp\" (UniqueName: \"kubernetes.io/projected/2dae1383-0ea3-4af6-adeb-097ea975a250-kube-api-access-47tfp\") pod \"authorino-76877994-pnlc9\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.416967 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.416946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rpfj4" Jul 15 19:09:16.417358 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.416945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.424001 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.423979 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rpfj4" Jul 15 19:09:16.426989 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.426971 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:16.496770 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.496735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/fc99a172-de85-46d6-ad47-0b35bb9a846c-kube-api-access-57wpr\") pod \"authorino-bcf6598c4-8z2ct\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.496943 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.496832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc99a172-de85-46d6-ad47-0b35bb9a846c-tls-cert\") pod \"authorino-bcf6598c4-8z2ct\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.499230 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.499203 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc99a172-de85-46d6-ad47-0b35bb9a846c-tls-cert\") pod \"authorino-bcf6598c4-8z2ct\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.504700 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.504681 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/fc99a172-de85-46d6-ad47-0b35bb9a846c-kube-api-access-57wpr\") pod \"authorino-bcf6598c4-8z2ct\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.589382 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.589345 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:09:16.597408 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.597385 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dae1383-0ea3-4af6-adeb-097ea975a250-tls-cert\") pod \"2dae1383-0ea3-4af6-adeb-097ea975a250\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " Jul 15 19:09:16.597514 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.597440 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sj5x\" (UniqueName: \"kubernetes.io/projected/96b2faaa-9c86-4109-a3b0-21ae6b4eb60d-kube-api-access-4sj5x\") pod \"96b2faaa-9c86-4109-a3b0-21ae6b4eb60d\" (UID: \"96b2faaa-9c86-4109-a3b0-21ae6b4eb60d\") " Jul 15 19:09:16.597560 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.597519 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47tfp\" (UniqueName: \"kubernetes.io/projected/2dae1383-0ea3-4af6-adeb-097ea975a250-kube-api-access-47tfp\") pod \"2dae1383-0ea3-4af6-adeb-097ea975a250\" (UID: \"2dae1383-0ea3-4af6-adeb-097ea975a250\") " Jul 15 19:09:16.599715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.599688 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dae1383-0ea3-4af6-adeb-097ea975a250-kube-api-access-47tfp" (OuterVolumeSpecName: "kube-api-access-47tfp") pod "2dae1383-0ea3-4af6-adeb-097ea975a250" (UID: "2dae1383-0ea3-4af6-adeb-097ea975a250"). InnerVolumeSpecName "kube-api-access-47tfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:09:16.599715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.599692 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dae1383-0ea3-4af6-adeb-097ea975a250-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2dae1383-0ea3-4af6-adeb-097ea975a250" (UID: "2dae1383-0ea3-4af6-adeb-097ea975a250"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:09:16.599855 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.599723 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b2faaa-9c86-4109-a3b0-21ae6b4eb60d-kube-api-access-4sj5x" (OuterVolumeSpecName: "kube-api-access-4sj5x") pod "96b2faaa-9c86-4109-a3b0-21ae6b4eb60d" (UID: "96b2faaa-9c86-4109-a3b0-21ae6b4eb60d"). InnerVolumeSpecName "kube-api-access-4sj5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:09:16.698809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.698778 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47tfp\" (UniqueName: \"kubernetes.io/projected/2dae1383-0ea3-4af6-adeb-097ea975a250-kube-api-access-47tfp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:09:16.698809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.698806 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dae1383-0ea3-4af6-adeb-097ea975a250-tls-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:09:16.698809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.698817 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sj5x\" (UniqueName: \"kubernetes.io/projected/96b2faaa-9c86-4109-a3b0-21ae6b4eb60d-kube-api-access-4sj5x\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:09:16.719953 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:16.719926 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcf6598c4-8z2ct"] Jul 15 19:09:16.721903 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:09:16.721865 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc99a172_de85_46d6_ad47_0b35bb9a846c.slice/crio-e1e93790b00b313bc041f375766c1df0c33a687fd19ade78b703ceb2b42f94e5 WatchSource:0}: Error finding container e1e93790b00b313bc041f375766c1df0c33a687fd19ade78b703ceb2b42f94e5: Status 404 returned error can't find the container with id e1e93790b00b313bc041f375766c1df0c33a687fd19ade78b703ceb2b42f94e5 Jul 15 19:09:17.422876 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.422843 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-pnlc9" Jul 15 19:09:17.422876 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.422865 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" event={"ID":"fc99a172-de85-46d6-ad47-0b35bb9a846c","Type":"ContainerStarted","Data":"32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc"} Jul 15 19:09:17.423371 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.422907 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" event={"ID":"fc99a172-de85-46d6-ad47-0b35bb9a846c","Type":"ContainerStarted","Data":"e1e93790b00b313bc041f375766c1df0c33a687fd19ade78b703ceb2b42f94e5"} Jul 15 19:09:17.423371 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.423037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rpfj4" Jul 15 19:09:17.439541 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.439491 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" podStartSLOduration=0.939541323 podStartE2EDuration="1.439472331s" totalImagesPullingTime="499.931008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:09:17.439322991 +0000 UTC m=+509.834678974" watchObservedRunningTime="2026-07-15 19:09:17.439472331 +0000 UTC m=+509.834828303" Jul 15 19:09:17.471351 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.471306 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-pnlc9"] Jul 15 19:09:17.482431 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.482397 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-pnlc9"] Jul 15 19:09:17.485203 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.485178 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wmqzj"] Jul 15 19:09:17.485606 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.485568 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-wmqzj" podUID="ee4af733-9e54-4115-9b69-9f1d7cb064ac" containerName="authorino" containerID="cri-o://dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459" gracePeriod=30 Jul 15 19:09:17.499738 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.499706 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rpfj4"] Jul 15 19:09:17.504196 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.504168 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rpfj4"] Jul 15 19:09:17.734582 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.734558 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wmqzj" Jul 15 19:09:17.908512 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.908480 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz444\" (UniqueName: \"kubernetes.io/projected/ee4af733-9e54-4115-9b69-9f1d7cb064ac-kube-api-access-lz444\") pod \"ee4af733-9e54-4115-9b69-9f1d7cb064ac\" (UID: \"ee4af733-9e54-4115-9b69-9f1d7cb064ac\") " Jul 15 19:09:17.910553 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:17.910519 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4af733-9e54-4115-9b69-9f1d7cb064ac-kube-api-access-lz444" (OuterVolumeSpecName: "kube-api-access-lz444") pod "ee4af733-9e54-4115-9b69-9f1d7cb064ac" (UID: "ee4af733-9e54-4115-9b69-9f1d7cb064ac"). InnerVolumeSpecName "kube-api-access-lz444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:09:18.009763 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.009679 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz444\" (UniqueName: \"kubernetes.io/projected/ee4af733-9e54-4115-9b69-9f1d7cb064ac-kube-api-access-lz444\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:09:18.198990 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.198956 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dae1383-0ea3-4af6-adeb-097ea975a250" path="/var/lib/kubelet/pods/2dae1383-0ea3-4af6-adeb-097ea975a250/volumes" Jul 15 19:09:18.199181 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.199168 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b2faaa-9c86-4109-a3b0-21ae6b4eb60d" path="/var/lib/kubelet/pods/96b2faaa-9c86-4109-a3b0-21ae6b4eb60d/volumes" Jul 15 19:09:18.429072 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.428992 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee4af733-9e54-4115-9b69-9f1d7cb064ac" containerID="dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459" exitCode=0 Jul 15 19:09:18.429072 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.429037 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wmqzj" Jul 15 19:09:18.429620 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.429073 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wmqzj" event={"ID":"ee4af733-9e54-4115-9b69-9f1d7cb064ac","Type":"ContainerDied","Data":"dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459"} Jul 15 19:09:18.429620 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.429115 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wmqzj" event={"ID":"ee4af733-9e54-4115-9b69-9f1d7cb064ac","Type":"ContainerDied","Data":"1d9f278dd179b907d22eca935d8b4c1273c1b8be7eebf73379fa7570279211c5"} Jul 15 19:09:18.429620 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.429133 2582 scope.go:117] "RemoveContainer" containerID="dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459" Jul 15 19:09:18.437671 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.437651 2582 scope.go:117] "RemoveContainer" containerID="dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459" Jul 15 19:09:18.437924 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:09:18.437905 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459\": container with ID starting with dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459 not found: ID does not exist" containerID="dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459" Jul 15 19:09:18.437991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.437934 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459"} err="failed to get container status \"dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459\": rpc error: code = NotFound desc = could not find container \"dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459\": container with ID starting with dd3c16db9ec280b778c47ffe2f8b231f753ddd4127154a4e8b788119a150e459 not found: ID does not exist" Jul 15 19:09:18.444475 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.444451 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wmqzj"] Jul 15 19:09:18.448289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:18.448267 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wmqzj"] Jul 15 19:09:20.199286 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:20.199232 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee4af733-9e54-4115-9b69-9f1d7cb064ac" path="/var/lib/kubelet/pods/ee4af733-9e54-4115-9b69-9f1d7cb064ac/volumes" Jul 15 19:09:22.362431 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.362397 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:22.376978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.376952 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b64cd75f7-8blp6"] Jul 15 19:09:22.377334 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.377320 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4af733-9e54-4115-9b69-9f1d7cb064ac" containerName="authorino" Jul 15 19:09:22.377386 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.377335 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4af733-9e54-4115-9b69-9f1d7cb064ac" containerName="authorino" Jul 15 19:09:22.377429 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.377393 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4af733-9e54-4115-9b69-9f1d7cb064ac" containerName="authorino" Jul 15 19:09:22.434427 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.434397 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b64cd75f7-8blp6"] Jul 15 19:09:22.434582 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.434530 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:22.438094 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.438075 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 19:09:22.438902 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.438882 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xtpdq\"" Jul 15 19:09:22.439229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.439201 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 19:09:22.464549 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.464486 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:22.529203 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.529165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:22.545665 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.545638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hjw\" (UniqueName: \"kubernetes.io/projected/79c82466-5374-49ff-bfa9-8572fe2ac792-kube-api-access-z4hjw\") pod \"maas-controller-6b64cd75f7-8blp6\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:22.545827 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.545706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert\") pod \"maas-controller-6b64cd75f7-8blp6\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:22.568531 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.568499 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d65cd49d-lxwsw"] Jul 15 19:09:22.591954 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.591930 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d65cd49d-lxwsw"] Jul 15 19:09:22.592082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.592026 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:22.646122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.646050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert\") pod \"maas-controller-6b64cd75f7-8blp6\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:22.646122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.646104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hjw\" (UniqueName: \"kubernetes.io/projected/79c82466-5374-49ff-bfa9-8572fe2ac792-kube-api-access-z4hjw\") pod \"maas-controller-6b64cd75f7-8blp6\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:22.646371 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:09:22.646214 2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 15 19:09:22.646371 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:09:22.646308 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert podName:79c82466-5374-49ff-bfa9-8572fe2ac792 nodeName:}" failed. No retries permitted until 2026-07-15 19:09:23.146286433 +0000 UTC m=+515.541642388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert") pod "maas-controller-6b64cd75f7-8blp6" (UID: "79c82466-5374-49ff-bfa9-8572fe2ac792") : secret "maas-controller-webhook-cert" not found Jul 15 19:09:22.655069 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.655044 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hjw\" (UniqueName: \"kubernetes.io/projected/79c82466-5374-49ff-bfa9-8572fe2ac792-kube-api-access-z4hjw\") pod \"maas-controller-6b64cd75f7-8blp6\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:22.746571 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.746539 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dff8c59e-06f2-4ab9-a296-994e43d0af8b-webhook-cert\") pod \"maas-controller-84d65cd49d-lxwsw\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:22.746740 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.746597 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfr4l\" (UniqueName: \"kubernetes.io/projected/dff8c59e-06f2-4ab9-a296-994e43d0af8b-kube-api-access-zfr4l\") pod \"maas-controller-84d65cd49d-lxwsw\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:22.770913 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.770881 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:22.847792 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.847760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfr4l\" (UniqueName: \"kubernetes.io/projected/dff8c59e-06f2-4ab9-a296-994e43d0af8b-kube-api-access-zfr4l\") pod \"maas-controller-84d65cd49d-lxwsw\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:22.847971 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.847864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dff8c59e-06f2-4ab9-a296-994e43d0af8b-webhook-cert\") pod \"maas-controller-84d65cd49d-lxwsw\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:22.850334 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.850312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dff8c59e-06f2-4ab9-a296-994e43d0af8b-webhook-cert\") pod \"maas-controller-84d65cd49d-lxwsw\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:22.856359 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.856335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfr4l\" (UniqueName: \"kubernetes.io/projected/dff8c59e-06f2-4ab9-a296-994e43d0af8b-kube-api-access-zfr4l\") pod \"maas-controller-84d65cd49d-lxwsw\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:22.902289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:22.902190 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:23.052498 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:23.052476 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d65cd49d-lxwsw"] Jul 15 19:09:23.054032 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:09:23.054007 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff8c59e_06f2_4ab9_a296_994e43d0af8b.slice/crio-3d31d7dffe2c1b5b6b8cbbef3be596556892f2ac489ff5a81faf73671e74282d WatchSource:0}: Error finding container 3d31d7dffe2c1b5b6b8cbbef3be596556892f2ac489ff5a81faf73671e74282d: Status 404 returned error can't find the container with id 3d31d7dffe2c1b5b6b8cbbef3be596556892f2ac489ff5a81faf73671e74282d Jul 15 19:09:23.151532 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:23.151497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert\") pod \"maas-controller-6b64cd75f7-8blp6\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:23.153912 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:23.153863 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert\") pod \"maas-controller-6b64cd75f7-8blp6\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:23.345053 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:23.345019 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:23.463772 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:23.463730 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" event={"ID":"dff8c59e-06f2-4ab9-a296-994e43d0af8b","Type":"ContainerStarted","Data":"3d31d7dffe2c1b5b6b8cbbef3be596556892f2ac489ff5a81faf73671e74282d"} Jul 15 19:09:23.484216 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:23.484185 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b64cd75f7-8blp6"] Jul 15 19:09:23.485828 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:09:23.485786 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c82466_5374_49ff_bfa9_8572fe2ac792.slice/crio-890caa61ecce22645caa135e9ec4313221857f63442a72c1cfd44e8883318e3c WatchSource:0}: Error finding container 890caa61ecce22645caa135e9ec4313221857f63442a72c1cfd44e8883318e3c: Status 404 returned error can't find the container with id 890caa61ecce22645caa135e9ec4313221857f63442a72c1cfd44e8883318e3c Jul 15 19:09:24.473710 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:24.473501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" event={"ID":"79c82466-5374-49ff-bfa9-8572fe2ac792","Type":"ContainerStarted","Data":"890caa61ecce22645caa135e9ec4313221857f63442a72c1cfd44e8883318e3c"} Jul 15 19:09:25.764512 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:25.764450 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:25.913041 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:25.912941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:26.486223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:26.486192 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" event={"ID":"dff8c59e-06f2-4ab9-a296-994e43d0af8b","Type":"ContainerStarted","Data":"90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6"} Jul 15 19:09:26.486434 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:26.486294 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:26.487853 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:26.487827 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" event={"ID":"79c82466-5374-49ff-bfa9-8572fe2ac792","Type":"ContainerStarted","Data":"605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c"} Jul 15 19:09:26.487973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:26.487919 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:26.510070 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:26.510024 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" podStartSLOduration=2.024758405 podStartE2EDuration="4.510009965s" totalImagesPullingTime="2.48525156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:09:26.508829574 +0000 UTC m=+518.904185545" watchObservedRunningTime="2026-07-15 19:09:26.510009965 +0000 UTC m=+518.905365935" Jul 15 19:09:26.537064 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:26.537008 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" podStartSLOduration=2.480043318 podStartE2EDuration="4.536992903s" totalImagesPullingTime="2.056949585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:09:26.535192392 +0000 UTC m=+518.930548362" watchObservedRunningTime="2026-07-15 19:09:26.536992903 +0000 UTC m=+518.932348872" Jul 15 19:09:32.923764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:32.923708 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:37.494503 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.494476 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:37.495005 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.494544 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:09:37.568232 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.568200 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b64cd75f7-8blp6"] Jul 15 19:09:37.568705 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.568673 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" podUID="79c82466-5374-49ff-bfa9-8572fe2ac792" containerName="manager" containerID="cri-o://605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c" gracePeriod=30 Jul 15 19:09:37.822661 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.822638 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:37.870936 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.870900 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert\") pod \"79c82466-5374-49ff-bfa9-8572fe2ac792\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " Jul 15 19:09:37.871107 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.870964 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4hjw\" (UniqueName: \"kubernetes.io/projected/79c82466-5374-49ff-bfa9-8572fe2ac792-kube-api-access-z4hjw\") pod \"79c82466-5374-49ff-bfa9-8572fe2ac792\" (UID: \"79c82466-5374-49ff-bfa9-8572fe2ac792\") " Jul 15 19:09:37.873292 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.873233 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "79c82466-5374-49ff-bfa9-8572fe2ac792" (UID: "79c82466-5374-49ff-bfa9-8572fe2ac792"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:09:37.873410 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.873299 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c82466-5374-49ff-bfa9-8572fe2ac792-kube-api-access-z4hjw" (OuterVolumeSpecName: "kube-api-access-z4hjw") pod "79c82466-5374-49ff-bfa9-8572fe2ac792" (UID: "79c82466-5374-49ff-bfa9-8572fe2ac792"). InnerVolumeSpecName "kube-api-access-z4hjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:09:37.971584 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.971549 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hjw\" (UniqueName: \"kubernetes.io/projected/79c82466-5374-49ff-bfa9-8572fe2ac792-kube-api-access-z4hjw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:09:37.971584 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:37.971578 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79c82466-5374-49ff-bfa9-8572fe2ac792-webhook-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:09:38.548766 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.548733 2582 generic.go:356] "Generic (PLEG): container finished" podID="79c82466-5374-49ff-bfa9-8572fe2ac792" containerID="605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c" exitCode=0 Jul 15 19:09:38.549209 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.548789 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" event={"ID":"79c82466-5374-49ff-bfa9-8572fe2ac792","Type":"ContainerDied","Data":"605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c"} Jul 15 19:09:38.549209 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.548794 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" Jul 15 19:09:38.549209 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.548834 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b64cd75f7-8blp6" event={"ID":"79c82466-5374-49ff-bfa9-8572fe2ac792","Type":"ContainerDied","Data":"890caa61ecce22645caa135e9ec4313221857f63442a72c1cfd44e8883318e3c"} Jul 15 19:09:38.549209 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.548860 2582 scope.go:117] "RemoveContainer" containerID="605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c" Jul 15 19:09:38.557761 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.557741 2582 scope.go:117] "RemoveContainer" containerID="605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c" Jul 15 19:09:38.558003 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:09:38.557986 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c\": container with ID starting with 605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c not found: ID does not exist" containerID="605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c" Jul 15 19:09:38.558071 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.558017 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c"} err="failed to get container status \"605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c\": rpc error: code = NotFound desc = could not find container \"605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c\": container with ID starting with 605589a989508a6a24e7fef691488c47170c3006f842501205358bebafc6a02c not found: ID does not exist" Jul 15 19:09:38.566963 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.566935 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b64cd75f7-8blp6"] Jul 15 19:09:38.571689 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:38.571666 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b64cd75f7-8blp6"] Jul 15 19:09:40.199107 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:40.199073 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c82466-5374-49ff-bfa9-8572fe2ac792" path="/var/lib/kubelet/pods/79c82466-5374-49ff-bfa9-8572fe2ac792/volumes" Jul 15 19:09:58.810007 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:58.809969 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:58.906883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:58.906852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:59.442577 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.442542 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:59.527309 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.527270 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd"] Jul 15 19:09:59.527689 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.527672 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c82466-5374-49ff-bfa9-8572fe2ac792" containerName="manager" Jul 15 19:09:59.527774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.527691 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c82466-5374-49ff-bfa9-8572fe2ac792" containerName="manager" Jul 15 19:09:59.527829 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.527798 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c82466-5374-49ff-bfa9-8572fe2ac792" containerName="manager" Jul 15 19:09:59.532145 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.532124 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.535525 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.535506 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bdsfz\"" Jul 15 19:09:59.535877 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.535863 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 15 19:09:59.545196 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.545178 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd"] Jul 15 19:09:59.568828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.568802 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:09:59.661486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.661453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-maas-api-tls\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.661486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.661489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-openshift-service-ca\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.661698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.661513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtgn\" (UniqueName: \"kubernetes.io/projected/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-kube-api-access-lrtgn\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.762675 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.762635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-maas-api-tls\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.762863 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.762687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-openshift-service-ca\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.762863 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.762714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtgn\" (UniqueName: \"kubernetes.io/projected/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-kube-api-access-lrtgn\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.763455 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.763434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-openshift-service-ca\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.765195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.765173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-maas-api-tls\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.771067 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.771049 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtgn\" (UniqueName: \"kubernetes.io/projected/ff216ef1-e553-40aa-8bc2-e96cb1117dcb-kube-api-access-lrtgn\") pod \"maas-api-7884b47f6b-jq4bd\" (UID: \"ff216ef1-e553-40aa-8bc2-e96cb1117dcb\") " pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.842566 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.842533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:09:59.983794 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:09:59.983770 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd"] Jul 15 19:09:59.986394 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:09:59.986363 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff216ef1_e553_40aa_8bc2_e96cb1117dcb.slice/crio-0e9b6fe26a1cf568c8572c92c494703bd0eec3a1f35c7acbda5e83f307cdb869 WatchSource:0}: Error finding container 0e9b6fe26a1cf568c8572c92c494703bd0eec3a1f35c7acbda5e83f307cdb869: Status 404 returned error can't find the container with id 0e9b6fe26a1cf568c8572c92c494703bd0eec3a1f35c7acbda5e83f307cdb869 Jul 15 19:10:00.154354 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.154274 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx"] Jul 15 19:10:00.157212 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.157195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.159897 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.159874 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 15 19:10:00.160011 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.159948 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lsnfb\"" Jul 15 19:10:00.171935 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.171913 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx"] Jul 15 19:10:00.267435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.267403 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbmv\" (UniqueName: \"kubernetes.io/projected/eb4f397e-cd77-4591-8885-db46cc7b0d0e-kube-api-access-lsbmv\") pod \"payload-pre-processing-74584bb9f8-nhxrx\" (UID: \"eb4f397e-cd77-4591-8885-db46cc7b0d0e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.267612 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.267459 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f397e-cd77-4591-8885-db46cc7b0d0e-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-nhxrx\" (UID: \"eb4f397e-cd77-4591-8885-db46cc7b0d0e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.302800 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.302759 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:00.368766 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.368732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbmv\" (UniqueName: \"kubernetes.io/projected/eb4f397e-cd77-4591-8885-db46cc7b0d0e-kube-api-access-lsbmv\") pod \"payload-pre-processing-74584bb9f8-nhxrx\" (UID: \"eb4f397e-cd77-4591-8885-db46cc7b0d0e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.368942 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.368787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f397e-cd77-4591-8885-db46cc7b0d0e-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-nhxrx\" (UID: \"eb4f397e-cd77-4591-8885-db46cc7b0d0e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.369487 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.369470 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f397e-cd77-4591-8885-db46cc7b0d0e-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-nhxrx\" (UID: \"eb4f397e-cd77-4591-8885-db46cc7b0d0e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.378782 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.378762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbmv\" (UniqueName: \"kubernetes.io/projected/eb4f397e-cd77-4591-8885-db46cc7b0d0e-kube-api-access-lsbmv\") pod \"payload-pre-processing-74584bb9f8-nhxrx\" (UID: \"eb4f397e-cd77-4591-8885-db46cc7b0d0e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.404009 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.403979 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:00.468300 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.468191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:00.601901 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.601871 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx"] Jul 15 19:10:00.605066 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:10:00.605038 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4f397e_cd77_4591_8885_db46cc7b0d0e.slice/crio-33974bdc29249ed68c3361f89061290d41901b2f079548aefc0e47c14926b95e WatchSource:0}: Error finding container 33974bdc29249ed68c3361f89061290d41901b2f079548aefc0e47c14926b95e: Status 404 returned error can't find the container with id 33974bdc29249ed68c3361f89061290d41901b2f079548aefc0e47c14926b95e Jul 15 19:10:00.662686 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.662648 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" event={"ID":"eb4f397e-cd77-4591-8885-db46cc7b0d0e","Type":"ContainerStarted","Data":"33974bdc29249ed68c3361f89061290d41901b2f079548aefc0e47c14926b95e"} Jul 15 19:10:00.663841 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:00.663820 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" event={"ID":"ff216ef1-e553-40aa-8bc2-e96cb1117dcb","Type":"ContainerStarted","Data":"0e9b6fe26a1cf568c8572c92c494703bd0eec3a1f35c7acbda5e83f307cdb869"} Jul 15 19:10:03.681324 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:03.681278 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" event={"ID":"ff216ef1-e553-40aa-8bc2-e96cb1117dcb","Type":"ContainerStarted","Data":"aff9193d038286c24f0be97d8453a639a00aebde2ca7627d14719ad440d385d7"} Jul 15 19:10:03.682162 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:03.682135 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:10:03.683955 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:03.683924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" event={"ID":"eb4f397e-cd77-4591-8885-db46cc7b0d0e","Type":"ContainerStarted","Data":"48f7ea4b8fa4b32e409246b1d04c2c4d9f3f240e8bef0cd2f1bf4afbf8e92746"} Jul 15 19:10:03.684476 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:03.684453 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:03.721017 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:03.720965 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" podStartSLOduration=1.619883481 podStartE2EDuration="3.720947833s" totalImagesPullingTime="2.101064352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:10:03.719563053 +0000 UTC m=+556.114919022" watchObservedRunningTime="2026-07-15 19:10:03.720947833 +0000 UTC m=+556.116303804" Jul 15 19:10:03.721487 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:03.721462 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" podStartSLOduration=1.998826448 podStartE2EDuration="4.721453462s" totalImagesPullingTime="2.722627014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:10:03.702453603 +0000 UTC m=+556.097809574" watchObservedRunningTime="2026-07-15 19:10:03.721453462 +0000 UTC m=+556.116809432" Jul 15 19:10:08.714916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:08.714819 2582 generic.go:356] "Generic (PLEG): container finished" podID="e67fea28-4b13-4985-9a30-b34c5460e8c8" containerID="4bbdafdd83cc406e0e4dd929a6ee23f44aca3a0bc6381b813e7bd43125277c87" exitCode=0 Jul 15 19:10:08.714916 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:08.714877 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" event={"ID":"e67fea28-4b13-4985-9a30-b34c5460e8c8","Type":"ContainerDied","Data":"4bbdafdd83cc406e0e4dd929a6ee23f44aca3a0bc6381b813e7bd43125277c87"} Jul 15 19:10:08.715395 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:08.715224 2582 scope.go:117] "RemoveContainer" containerID="4bbdafdd83cc406e0e4dd929a6ee23f44aca3a0bc6381b813e7bd43125277c87" Jul 15 19:10:08.716180 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:08.716150 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:10:08.716232 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:08.716224 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:10:08.716294 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:08.716277 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:10:09.723370 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:09.723338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" event={"ID":"e67fea28-4b13-4985-9a30-b34c5460e8c8","Type":"ContainerStarted","Data":"8df9f48f57bbdabd85abfe53f5fd3a082a95756714d891ca83a1efcc73645156"} Jul 15 19:10:10.090319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:10.090209 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:10:10.091060 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:10.091036 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-79zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:10:10.091180 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:10.091083 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" podUID="e67fea28-4b13-4985-9a30-b34c5460e8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:10:10.707414 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:10.707384 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7884b47f6b-jq4bd" Jul 15 19:10:11.090520 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:11.090471 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-79zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:10:11.091028 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:11.090544 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" podUID="e67fea28-4b13-4985-9a30-b34c5460e8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:10:11.695173 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:11.695131 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:11.788717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:11.788687 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:12.090009 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:12.089965 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-79zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:10:12.090227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:12.090036 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" podUID="e67fea28-4b13-4985-9a30-b34c5460e8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:10:12.215075 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:12.215041 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:10:13.090626 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:13.090583 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-79zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:10:13.090842 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:13.090657 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" podUID="e67fea28-4b13-4985-9a30-b34c5460e8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:10:14.090602 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:14.090563 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-79zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:10:14.091176 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:14.091146 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" podUID="e67fea28-4b13-4985-9a30-b34c5460e8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:10:15.094186 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:15.094154 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:10:15.095309 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:15.095289 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-79zpb" Jul 15 19:10:15.699414 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:15.699386 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-nhxrx" Jul 15 19:10:20.755265 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.755211 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w"] Jul 15 19:10:20.758787 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.758769 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.762509 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.762327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6xgvb\"" Jul 15 19:10:20.763361 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.763314 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 19:10:20.763459 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.763431 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 19:10:20.763694 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.763678 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 19:10:20.768993 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.768965 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.769100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.768993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.769100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.769029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.769215 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.769112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd49h\" (UniqueName: \"kubernetes.io/projected/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-kube-api-access-pd49h\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.769215 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.769157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.769215 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.769180 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.770915 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.770890 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w"] Jul 15 19:10:20.785719 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.785681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:20.870464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.870402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.870464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.870466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd49h\" (UniqueName: \"kubernetes.io/projected/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-kube-api-access-pd49h\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.870732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.870518 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.870732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.870554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.870732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.870618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.870732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.870639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.872132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.871614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.872132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.871708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.872132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.871974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.873822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.873796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.874357 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.874340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:20.879062 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:20.879043 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd49h\" (UniqueName: \"kubernetes.io/projected/57c97fa1-8ffd-414b-afff-a2a9eb725dbd-kube-api-access-pd49h\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w\" (UID: \"57c97fa1-8ffd-414b-afff-a2a9eb725dbd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:21.071313 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:21.071199 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:21.214478 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:21.214447 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w"] Jul 15 19:10:21.215603 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:10:21.215578 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c97fa1_8ffd_414b_afff_a2a9eb725dbd.slice/crio-a00fff1c2474e3a246bb41199e1e1cfdd7e82aa8f2ab2a5f148fa360d00decc9 WatchSource:0}: Error finding container a00fff1c2474e3a246bb41199e1e1cfdd7e82aa8f2ab2a5f148fa360d00decc9: Status 404 returned error can't find the container with id a00fff1c2474e3a246bb41199e1e1cfdd7e82aa8f2ab2a5f148fa360d00decc9 Jul 15 19:10:21.792188 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:21.792143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" event={"ID":"57c97fa1-8ffd-414b-afff-a2a9eb725dbd","Type":"ContainerStarted","Data":"a00fff1c2474e3a246bb41199e1e1cfdd7e82aa8f2ab2a5f148fa360d00decc9"} Jul 15 19:10:22.798341 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:22.798303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" event={"ID":"57c97fa1-8ffd-414b-afff-a2a9eb725dbd","Type":"ContainerStarted","Data":"9e43f57860009615c940ccbaa96a9afc3b9e8e32398ecc82d7b938d386637c47"} Jul 15 19:10:22.817132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:22.817081 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" podStartSLOduration=1.595737749 podStartE2EDuration="2.817064866s" totalImagesPullingTime="1.221327117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:10:22.814870233 +0000 UTC m=+575.210226243" watchObservedRunningTime="2026-07-15 19:10:22.817064866 +0000 UTC m=+575.212420835" Jul 15 19:10:24.263063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:24.263023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:31.072299 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:31.072264 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:31.072831 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:31.072348 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:31.084999 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:31.084975 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:31.853912 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:31.853883 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w" Jul 15 19:10:35.270382 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:35.270339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:38.005721 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:38.005686 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:41.126037 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.126008 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc"] Jul 15 19:10:41.131225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.131204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.133663 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.133644 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 19:10:41.140526 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.140501 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc"] Jul 15 19:10:41.155560 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.155541 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:41.251625 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.251594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99896c41-20e5-4929-aece-3d5cf8a1a2cc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.251811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.251638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.251811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.251677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.251811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.251765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.251811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.251801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.251982 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.251865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5d87\" (UniqueName: \"kubernetes.io/projected/99896c41-20e5-4929-aece-3d5cf8a1a2cc-kube-api-access-n5d87\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.352639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.352605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99896c41-20e5-4929-aece-3d5cf8a1a2cc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.352828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.352654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.352828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.352688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.352828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.352720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.352828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.352745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.352828 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.352762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5d87\" (UniqueName: \"kubernetes.io/projected/99896c41-20e5-4929-aece-3d5cf8a1a2cc-kube-api-access-n5d87\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.353192 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.353168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.353297 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.353205 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.353297 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.353278 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.354985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.354964 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99896c41-20e5-4929-aece-3d5cf8a1a2cc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.355111 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.355094 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99896c41-20e5-4929-aece-3d5cf8a1a2cc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.362183 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.362159 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5d87\" (UniqueName: \"kubernetes.io/projected/99896c41-20e5-4929-aece-3d5cf8a1a2cc-kube-api-access-n5d87\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-stxcc\" (UID: \"99896c41-20e5-4929-aece-3d5cf8a1a2cc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.442552 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.442478 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:41.579101 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.579078 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc"] Jul 15 19:10:41.581406 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:10:41.581366 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99896c41_20e5_4929_aece_3d5cf8a1a2cc.slice/crio-c63f75f3c699b4216bd74ac666acbebdb7a3bc863dd6f6bc3203053eb2eea072 WatchSource:0}: Error finding container c63f75f3c699b4216bd74ac666acbebdb7a3bc863dd6f6bc3203053eb2eea072: Status 404 returned error can't find the container with id c63f75f3c699b4216bd74ac666acbebdb7a3bc863dd6f6bc3203053eb2eea072 Jul 15 19:10:41.897619 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:41.897593 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" event={"ID":"99896c41-20e5-4929-aece-3d5cf8a1a2cc","Type":"ContainerStarted","Data":"c63f75f3c699b4216bd74ac666acbebdb7a3bc863dd6f6bc3203053eb2eea072"} Jul 15 19:10:42.903665 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:42.903625 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" event={"ID":"99896c41-20e5-4929-aece-3d5cf8a1a2cc","Type":"ContainerStarted","Data":"0c5fbf62240e679151fc37e2951c9ac071ac144a07a03c606589f259a05c7f8b"} Jul 15 19:10:42.930251 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:42.930179 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" podStartSLOduration=1.644625016 podStartE2EDuration="1.930163856s" totalImagesPullingTime="285.53884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:10:42.927002546 +0000 UTC m=+595.322358515" watchObservedRunningTime="2026-07-15 19:10:42.930163856 +0000 UTC m=+595.325519827" Jul 15 19:10:48.069814 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:48.069787 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:10:48.070212 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:48.070144 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:10:51.443081 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:51.443043 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:51.443620 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:51.443127 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:51.455643 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:51.455621 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:10:51.814308 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:51.814254 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:10:51.966395 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:10:51.966369 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-stxcc" Jul 15 19:11:09.077483 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:11:09.077444 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:11:09.137029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:11:09.136999 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:11:39.888509 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:11:39.888475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:12:03.901808 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:03.901773 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7999c8f867-kkgcp"] Jul 15 19:12:03.905540 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:03.905519 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:03.911646 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:03.911617 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7999c8f867-kkgcp"] Jul 15 19:12:03.953640 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:03.953617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46e5766e-2a33-4d5d-a45f-0cbbfa86325c-tls-cert\") pod \"authorino-7999c8f867-kkgcp\" (UID: \"46e5766e-2a33-4d5d-a45f-0cbbfa86325c\") " pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:03.953746 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:03.953651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p778p\" (UniqueName: \"kubernetes.io/projected/46e5766e-2a33-4d5d-a45f-0cbbfa86325c-kube-api-access-p778p\") pod \"authorino-7999c8f867-kkgcp\" (UID: \"46e5766e-2a33-4d5d-a45f-0cbbfa86325c\") " pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:04.054714 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.054674 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46e5766e-2a33-4d5d-a45f-0cbbfa86325c-tls-cert\") pod \"authorino-7999c8f867-kkgcp\" (UID: \"46e5766e-2a33-4d5d-a45f-0cbbfa86325c\") " pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:04.054887 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.054725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p778p\" (UniqueName: \"kubernetes.io/projected/46e5766e-2a33-4d5d-a45f-0cbbfa86325c-kube-api-access-p778p\") pod \"authorino-7999c8f867-kkgcp\" (UID: \"46e5766e-2a33-4d5d-a45f-0cbbfa86325c\") " pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:04.057098 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.057073 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46e5766e-2a33-4d5d-a45f-0cbbfa86325c-tls-cert\") pod \"authorino-7999c8f867-kkgcp\" (UID: \"46e5766e-2a33-4d5d-a45f-0cbbfa86325c\") " pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:04.062663 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.062639 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p778p\" (UniqueName: \"kubernetes.io/projected/46e5766e-2a33-4d5d-a45f-0cbbfa86325c-kube-api-access-p778p\") pod \"authorino-7999c8f867-kkgcp\" (UID: \"46e5766e-2a33-4d5d-a45f-0cbbfa86325c\") " pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:04.214672 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.214600 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7999c8f867-kkgcp" Jul 15 19:12:04.301458 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.300838 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:12:04.355996 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.355972 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7999c8f867-kkgcp"] Jul 15 19:12:04.357794 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:12:04.357764 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e5766e_2a33_4d5d_a45f_0cbbfa86325c.slice/crio-8440f65925a77c0628aafbbda55108fa02ae3f7958f6f710f843315196dfd33e WatchSource:0}: Error finding container 8440f65925a77c0628aafbbda55108fa02ae3f7958f6f710f843315196dfd33e: Status 404 returned error can't find the container with id 8440f65925a77c0628aafbbda55108fa02ae3f7958f6f710f843315196dfd33e Jul 15 19:12:04.359105 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.359087 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:12:04.395883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:04.395854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:12:05.328913 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.328865 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7999c8f867-kkgcp" event={"ID":"46e5766e-2a33-4d5d-a45f-0cbbfa86325c","Type":"ContainerStarted","Data":"e02d47bce02209bfd286b1e7bffbe05ddd10fd73a99e2aabd7be853b58d7ae65"} Jul 15 19:12:05.329415 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.328930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7999c8f867-kkgcp" event={"ID":"46e5766e-2a33-4d5d-a45f-0cbbfa86325c","Type":"ContainerStarted","Data":"8440f65925a77c0628aafbbda55108fa02ae3f7958f6f710f843315196dfd33e"} Jul 15 19:12:05.399434 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.396988 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7999c8f867-kkgcp" podStartSLOduration=2.055452014 podStartE2EDuration="2.396965769s" totalImagesPullingTime="341.513755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:12:05.352382889 +0000 UTC m=+677.747738860" watchObservedRunningTime="2026-07-15 19:12:05.396965769 +0000 UTC m=+677.792321740" Jul 15 19:12:05.399434 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.398280 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bcf6598c4-8z2ct"] Jul 15 19:12:05.399434 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.398484 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" podUID="fc99a172-de85-46d6-ad47-0b35bb9a846c" containerName="authorino" containerID="cri-o://32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc" gracePeriod=30 Jul 15 19:12:05.524425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.524326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:12:05.621072 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.620967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:12:05.667667 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.667632 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:12:05.668028 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.668005 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc99a172-de85-46d6-ad47-0b35bb9a846c-tls-cert\") pod \"fc99a172-de85-46d6-ad47-0b35bb9a846c\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " Jul 15 19:12:05.668129 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.668055 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/fc99a172-de85-46d6-ad47-0b35bb9a846c-kube-api-access-57wpr\") pod \"fc99a172-de85-46d6-ad47-0b35bb9a846c\" (UID: \"fc99a172-de85-46d6-ad47-0b35bb9a846c\") " Jul 15 19:12:05.670182 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.670149 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc99a172-de85-46d6-ad47-0b35bb9a846c-kube-api-access-57wpr" (OuterVolumeSpecName: "kube-api-access-57wpr") pod "fc99a172-de85-46d6-ad47-0b35bb9a846c" (UID: "fc99a172-de85-46d6-ad47-0b35bb9a846c"). InnerVolumeSpecName "kube-api-access-57wpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:12:05.681401 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.681364 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc99a172-de85-46d6-ad47-0b35bb9a846c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fc99a172-de85-46d6-ad47-0b35bb9a846c" (UID: "fc99a172-de85-46d6-ad47-0b35bb9a846c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:12:05.769299 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.769235 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc99a172-de85-46d6-ad47-0b35bb9a846c-tls-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:12:05.769299 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:05.769299 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/fc99a172-de85-46d6-ad47-0b35bb9a846c-kube-api-access-57wpr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:12:06.339369 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.339282 2582 generic.go:356] "Generic (PLEG): container finished" podID="fc99a172-de85-46d6-ad47-0b35bb9a846c" containerID="32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc" exitCode=0 Jul 15 19:12:06.340453 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.339883 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" Jul 15 19:12:06.343943 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.339903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" event={"ID":"fc99a172-de85-46d6-ad47-0b35bb9a846c","Type":"ContainerDied","Data":"32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc"} Jul 15 19:12:06.344119 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.344097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcf6598c4-8z2ct" event={"ID":"fc99a172-de85-46d6-ad47-0b35bb9a846c","Type":"ContainerDied","Data":"e1e93790b00b313bc041f375766c1df0c33a687fd19ade78b703ceb2b42f94e5"} Jul 15 19:12:06.344231 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.344218 2582 scope.go:117] "RemoveContainer" containerID="32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc" Jul 15 19:12:06.365641 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.365615 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bcf6598c4-8z2ct"] Jul 15 19:12:06.369045 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.368973 2582 scope.go:117] "RemoveContainer" containerID="32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc" Jul 15 19:12:06.369347 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:12:06.369321 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc\": container with ID starting with 32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc not found: ID does not exist" containerID="32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc" Jul 15 19:12:06.369438 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.369356 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc"} err="failed to get container status \"32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc\": rpc error: code = NotFound desc = could not find container \"32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc\": container with ID starting with 32b991c5d713571c312a7302a1863ae6691b5564acb4a5dc94c24f05675a9dcc not found: ID does not exist" Jul 15 19:12:06.370972 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:06.370945 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bcf6598c4-8z2ct"] Jul 15 19:12:08.199319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:12:08.199281 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc99a172-de85-46d6-ad47-0b35bb9a846c" path="/var/lib/kubelet/pods/fc99a172-de85-46d6-ad47-0b35bb9a846c/volumes" Jul 15 19:13:19.760399 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:19.760360 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:13:19.850684 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:19.850654 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d65cd49d-lxwsw"] Jul 15 19:13:19.850880 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:19.850853 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" podUID="dff8c59e-06f2-4ab9-a296-994e43d0af8b" containerName="manager" containerID="cri-o://90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6" gracePeriod=30 Jul 15 19:13:20.101891 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.101866 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:13:20.214946 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.214920 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dff8c59e-06f2-4ab9-a296-994e43d0af8b-webhook-cert\") pod \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " Jul 15 19:13:20.215137 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.214957 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfr4l\" (UniqueName: \"kubernetes.io/projected/dff8c59e-06f2-4ab9-a296-994e43d0af8b-kube-api-access-zfr4l\") pod \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\" (UID: \"dff8c59e-06f2-4ab9-a296-994e43d0af8b\") " Jul 15 19:13:20.217007 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.216981 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff8c59e-06f2-4ab9-a296-994e43d0af8b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dff8c59e-06f2-4ab9-a296-994e43d0af8b" (UID: "dff8c59e-06f2-4ab9-a296-994e43d0af8b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:13:20.217121 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.217050 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff8c59e-06f2-4ab9-a296-994e43d0af8b-kube-api-access-zfr4l" (OuterVolumeSpecName: "kube-api-access-zfr4l") pod "dff8c59e-06f2-4ab9-a296-994e43d0af8b" (UID: "dff8c59e-06f2-4ab9-a296-994e43d0af8b"). InnerVolumeSpecName "kube-api-access-zfr4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:13:20.315804 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.315727 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dff8c59e-06f2-4ab9-a296-994e43d0af8b-webhook-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:13:20.315804 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.315751 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfr4l\" (UniqueName: \"kubernetes.io/projected/dff8c59e-06f2-4ab9-a296-994e43d0af8b-kube-api-access-zfr4l\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:13:20.726602 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.726571 2582 generic.go:356] "Generic (PLEG): container finished" podID="dff8c59e-06f2-4ab9-a296-994e43d0af8b" containerID="90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6" exitCode=0 Jul 15 19:13:20.726762 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.726610 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" Jul 15 19:13:20.726762 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.726651 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" event={"ID":"dff8c59e-06f2-4ab9-a296-994e43d0af8b","Type":"ContainerDied","Data":"90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6"} Jul 15 19:13:20.726762 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.726703 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d65cd49d-lxwsw" event={"ID":"dff8c59e-06f2-4ab9-a296-994e43d0af8b","Type":"ContainerDied","Data":"3d31d7dffe2c1b5b6b8cbbef3be596556892f2ac489ff5a81faf73671e74282d"} Jul 15 19:13:20.726762 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.726730 2582 scope.go:117] "RemoveContainer" containerID="90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6" Jul 15 19:13:20.737384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.737363 2582 scope.go:117] "RemoveContainer" containerID="90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6" Jul 15 19:13:20.737703 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:13:20.737677 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6\": container with ID starting with 90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6 not found: ID does not exist" containerID="90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6" Jul 15 19:13:20.737783 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.737714 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6"} err="failed to get container status \"90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6\": rpc error: code = NotFound desc = could not find container \"90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6\": container with ID starting with 90a9e5f106ce51747101f2dd7a9136694356fd4d338e7629f8b37413730019d6 not found: ID does not exist" Jul 15 19:13:20.748795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.748766 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d65cd49d-lxwsw"] Jul 15 19:13:20.753558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:20.753537 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84d65cd49d-lxwsw"] Jul 15 19:13:21.004840 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.004741 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:13:21.067447 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.067412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:13:21.421661 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.421569 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:13:21.536826 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.536792 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d65cd49d-zsvwc"] Jul 15 19:13:21.537162 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.537150 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff8c59e-06f2-4ab9-a296-994e43d0af8b" containerName="manager" Jul 15 19:13:21.537208 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.537164 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff8c59e-06f2-4ab9-a296-994e43d0af8b" containerName="manager" Jul 15 19:13:21.537208 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.537179 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc99a172-de85-46d6-ad47-0b35bb9a846c" containerName="authorino" Jul 15 19:13:21.537208 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.537184 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc99a172-de85-46d6-ad47-0b35bb9a846c" containerName="authorino" Jul 15 19:13:21.537321 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.537253 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff8c59e-06f2-4ab9-a296-994e43d0af8b" containerName="manager" Jul 15 19:13:21.537321 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.537271 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc99a172-de85-46d6-ad47-0b35bb9a846c" containerName="authorino" Jul 15 19:13:21.541617 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.541598 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.546510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.546492 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 19:13:21.546665 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.546646 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xtpdq\"" Jul 15 19:13:21.546719 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.546664 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 19:13:21.557370 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.557349 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d65cd49d-zsvwc"] Jul 15 19:13:21.624984 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.624949 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hvd\" (UniqueName: \"kubernetes.io/projected/d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1-kube-api-access-68hvd\") pod \"maas-controller-84d65cd49d-zsvwc\" (UID: \"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1\") " pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.625159 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.624990 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1-webhook-cert\") pod \"maas-controller-84d65cd49d-zsvwc\" (UID: \"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1\") " pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.726517 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.726483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hvd\" (UniqueName: \"kubernetes.io/projected/d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1-kube-api-access-68hvd\") pod \"maas-controller-84d65cd49d-zsvwc\" (UID: \"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1\") " pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.726708 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.726532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1-webhook-cert\") pod \"maas-controller-84d65cd49d-zsvwc\" (UID: \"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1\") " pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.728990 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.728965 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1-webhook-cert\") pod \"maas-controller-84d65cd49d-zsvwc\" (UID: \"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1\") " pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.738284 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.738256 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hvd\" (UniqueName: \"kubernetes.io/projected/d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1-kube-api-access-68hvd\") pod \"maas-controller-84d65cd49d-zsvwc\" (UID: \"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1\") " pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.851809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.851778 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:21.986570 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:21.986544 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d65cd49d-zsvwc"] Jul 15 19:13:21.987949 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:13:21.987923 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d4fa3b_c102_495f_b0e8_7fc80e38e8d1.slice/crio-3e22dc5fadc1425b66ee7ac165c0e5dbfa78545bf3811217990f61ad6426cbc9 WatchSource:0}: Error finding container 3e22dc5fadc1425b66ee7ac165c0e5dbfa78545bf3811217990f61ad6426cbc9: Status 404 returned error can't find the container with id 3e22dc5fadc1425b66ee7ac165c0e5dbfa78545bf3811217990f61ad6426cbc9 Jul 15 19:13:22.200230 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:22.200196 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff8c59e-06f2-4ab9-a296-994e43d0af8b" path="/var/lib/kubelet/pods/dff8c59e-06f2-4ab9-a296-994e43d0af8b/volumes" Jul 15 19:13:22.740578 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:22.740539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d65cd49d-zsvwc" event={"ID":"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1","Type":"ContainerStarted","Data":"b40e570d195bb18c011c3b0529a7d9ebcd05c4641412920ee88b6d5b25c13abb"} Jul 15 19:13:22.740732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:22.740591 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d65cd49d-zsvwc" event={"ID":"d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1","Type":"ContainerStarted","Data":"3e22dc5fadc1425b66ee7ac165c0e5dbfa78545bf3811217990f61ad6426cbc9"} Jul 15 19:13:22.740732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:22.740620 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:22.792257 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:22.792135 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d65cd49d-zsvwc" podStartSLOduration=1.285410122 podStartE2EDuration="1.792117583s" totalImagesPullingTime="506.707461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:13:22.788480328 +0000 UTC m=+755.183836324" watchObservedRunningTime="2026-07-15 19:13:22.792117583 +0000 UTC m=+755.187473552" Jul 15 19:13:33.749046 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:33.749016 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d65cd49d-zsvwc" Jul 15 19:13:57.003123 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:57.003084 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:13:57.061649 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:13:57.061611 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:02.350952 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:14:02.350915 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:02.462838 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:14:02.462804 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:08.728450 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:14:08.728410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:08.828554 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:14:08.828521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:14.990035 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:14:14.989944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:15.063342 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:14:15.063306 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:16.041412 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:15:16.041378 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:16.110533 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:15:16.110505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.114815 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:15:48.114781 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:15:48.115406 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:15:48.115351 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:15:53.945379 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:15:53.945340 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:55.149659 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:15:55.149627 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:29.363734 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:20:29.363707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:29.431260 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:20:29.431216 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:48.146165 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:20:48.146137 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:20:48.147932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:20:48.147901 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:21:01.859638 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:21:01.859607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:36.368888 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:21:36.368852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:53.137963 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:21:53.137929 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:13.913486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:13.913445 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:21.450715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:21.450678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:30.543907 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:30.543871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:34.884592 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:34.884556 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:37.827515 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:37.827476 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:37.915718 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:37.915678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:38.292449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:38.292415 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:44.874914 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:44.874875 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:45.173825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.173799 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:45.204445 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.204412 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb"] Jul 15 19:25:45.207969 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.207952 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.210678 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.210663 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-62375b76-openshift-default-dockercfg-fqc8p\"" Jul 15 19:25:45.246900 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.246867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-socket\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247056 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.246905 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-data\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247056 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.246982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-envoy\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.247063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28c63-7d68-429c-893d-25f7f001958e-istiod-ca-cert\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.247129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqq4v\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-kube-api-access-wqq4v\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.247179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-credential-socket\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247342 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.247205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28c63-7d68-429c-893d-25f7f001958e-istio-podinfo\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247342 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.247231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-istio-token\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.247342 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.247319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-certs\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.348510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-data\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.348510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-envoy\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.348784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28c63-7d68-429c-893d-25f7f001958e-istiod-ca-cert\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.348784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqq4v\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-kube-api-access-wqq4v\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.348784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-credential-socket\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.348784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28c63-7d68-429c-893d-25f7f001958e-istio-podinfo\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.348784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-istio-token\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.349065 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-certs\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.349065 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.348892 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-socket\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.349065 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.349043 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-data\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.349273 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.349158 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-credential-socket\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.349273 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.349224 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-socket\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.349398 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.349361 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-certs\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.349455 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.349407 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28c63-7d68-429c-893d-25f7f001958e-istiod-ca-cert\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.351500 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.351475 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-envoy\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.352097 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.352078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28c63-7d68-429c-893d-25f7f001958e-istio-podinfo\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.358922 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.358896 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb"] Jul 15 19:25:45.362060 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.362016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:45.372913 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.372883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-istio-token\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.373812 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.373790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqq4v\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-kube-api-access-wqq4v\") pod \"e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.468285 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.467913 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8"] Jul 15 19:25:45.472987 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.472962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.478228 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.478195 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:45.478413 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.478281 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-62375b76-serving-cert\"" Jul 15 19:25:45.492785 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.492754 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8"] Jul 15 19:25:45.518191 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.518158 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:45.551307 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.551277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.551473 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.551326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpsw7\" (UniqueName: \"kubernetes.io/projected/4506247d-66eb-4271-86a3-95a33ccfcc40-kube-api-access-qpsw7\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.551473 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.551379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4506247d-66eb-4271-86a3-95a33ccfcc40-openshift-service-ca\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.652829 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.652786 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4506247d-66eb-4271-86a3-95a33ccfcc40-openshift-service-ca\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.653001 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.652922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.653001 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.652956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpsw7\" (UniqueName: \"kubernetes.io/projected/4506247d-66eb-4271-86a3-95a33ccfcc40-kube-api-access-qpsw7\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.653093 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:25:45.653071 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-serving-cert: secret "maas-api-e2e-ait-62375b76-serving-cert" not found Jul 15 19:25:45.653176 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:25:45.653161 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls podName:4506247d-66eb-4271-86a3-95a33ccfcc40 nodeName:}" failed. No retries permitted until 2026-07-15 19:25:46.153138522 +0000 UTC m=+1498.548494479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls") pod "maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" (UID: "4506247d-66eb-4271-86a3-95a33ccfcc40") : secret "maas-api-e2e-ait-62375b76-serving-cert" not found Jul 15 19:25:45.653601 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.653575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4506247d-66eb-4271-86a3-95a33ccfcc40-openshift-service-ca\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.684004 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.683966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpsw7\" (UniqueName: \"kubernetes.io/projected/4506247d-66eb-4271-86a3-95a33ccfcc40-kube-api-access-qpsw7\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:45.696911 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.696885 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:45.708594 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.708573 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb"] Jul 15 19:25:45.710313 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:25:45.710289 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba28c63_7d68_429c_893d_25f7f001958e.slice/crio-ac51280dd6b709baff76e146796541204b061d77bcf99c950a91b65b94434d8c WatchSource:0}: Error finding container ac51280dd6b709baff76e146796541204b061d77bcf99c950a91b65b94434d8c: Status 404 returned error can't find the container with id ac51280dd6b709baff76e146796541204b061d77bcf99c950a91b65b94434d8c Jul 15 19:25:45.712629 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.712594 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:25:45.712714 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.712657 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:25:45.712714 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.712690 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:25:45.947384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:45.947354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:46.157737 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.157504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:46.160015 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.159985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls\") pod \"maas-api-e2e-ait-62375b76-6484464c8f-lgsv8\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:46.350801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.350769 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:46.386469 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.386434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:46.453129 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.453098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:46.615966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.615927 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" event={"ID":"8ba28c63-7d68-429c-893d-25f7f001958e","Type":"ContainerStarted","Data":"bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d"} Jul 15 19:25:46.616150 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.615988 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" event={"ID":"8ba28c63-7d68-429c-893d-25f7f001958e","Type":"ContainerStarted","Data":"ac51280dd6b709baff76e146796541204b061d77bcf99c950a91b65b94434d8c"} Jul 15 19:25:46.646043 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.645997 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" podStartSLOduration=1.645982952 podStartE2EDuration="1.645982952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:25:46.641830772 +0000 UTC m=+1499.037186742" watchObservedRunningTime="2026-07-15 19:25:46.645982952 +0000 UTC m=+1499.041338921" Jul 15 19:25:46.738220 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.738195 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8"] Jul 15 19:25:46.739147 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:25:46.739123 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4506247d_66eb_4271_86a3_95a33ccfcc40.slice/crio-3ce0b63f194741e323ff8d3597bdf11a6298ec96a902c127351644aaa3a6324b WatchSource:0}: Error finding container 3ce0b63f194741e323ff8d3597bdf11a6298ec96a902c127351644aaa3a6324b: Status 404 returned error can't find the container with id 3ce0b63f194741e323ff8d3597bdf11a6298ec96a902c127351644aaa3a6324b Jul 15 19:25:46.740618 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:46.740596 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:25:47.518320 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:47.518271 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:47.523417 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:47.523389 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:47.622633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:47.622590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" event={"ID":"4506247d-66eb-4271-86a3-95a33ccfcc40","Type":"ContainerStarted","Data":"f5a4c631cfda634aad45ddb8997323eb84a4f0ae5cd01515c8d7b36652d6aebc"} Jul 15 19:25:47.622821 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:47.622645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" event={"ID":"4506247d-66eb-4271-86a3-95a33ccfcc40","Type":"ContainerStarted","Data":"3ce0b63f194741e323ff8d3597bdf11a6298ec96a902c127351644aaa3a6324b"} Jul 15 19:25:47.622882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:47.622844 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:47.623837 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:47.623817 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:47.653119 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:47.653076 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" podStartSLOduration=2.241132412 podStartE2EDuration="2.653062021s" totalImagesPullingTime="411.929609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:25:47.652157609 +0000 UTC m=+1500.047513580" watchObservedRunningTime="2026-07-15 19:25:47.653062021 +0000 UTC m=+1500.048417991" Jul 15 19:25:48.179470 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:48.179443 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:25:48.184563 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:48.184541 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:25:48.635528 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:48.635494 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:49.907014 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:49.906985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:51.413930 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:51.413895 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb"] Jul 15 19:25:51.414344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:51.414103 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" podUID="8ba28c63-7d68-429c-893d-25f7f001958e" containerName="istio-proxy" containerID="cri-o://bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d" gracePeriod=30 Jul 15 19:25:51.471964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:51.471935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:51.590153 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:51.590113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:52.904266 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:52.900749 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:54.646903 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:54.646826 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:57.270086 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.270058 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:57.364511 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364431 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqq4v\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-kube-api-access-wqq4v\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364511 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364481 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-certs\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364703 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364518 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28c63-7d68-429c-893d-25f7f001958e-istiod-ca-cert\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364703 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364582 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-data\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364703 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364617 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-envoy\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364703 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364654 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28c63-7d68-429c-893d-25f7f001958e-istio-podinfo\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364703 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364700 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-socket\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364970 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364739 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-credential-socket\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364970 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-istio-token\") pod \"8ba28c63-7d68-429c-893d-25f7f001958e\" (UID: \"8ba28c63-7d68-429c-893d-25f7f001958e\") " Jul 15 19:25:57.364970 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364861 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:25:57.365145 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.364967 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba28c63-7d68-429c-893d-25f7f001958e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:25:57.365208 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.365191 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.365277 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.365214 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28c63-7d68-429c-893d-25f7f001958e-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.365343 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.365267 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:25:57.365421 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.365391 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:25:57.366087 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.366044 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:25:57.367039 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.367004 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:25:57.367039 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.367025 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:25:57.367219 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.367081 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba28c63-7d68-429c-893d-25f7f001958e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:25:57.367306 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.367284 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-kube-api-access-wqq4v" (OuterVolumeSpecName: "kube-api-access-wqq4v") pod "8ba28c63-7d68-429c-893d-25f7f001958e" (UID: "8ba28c63-7d68-429c-893d-25f7f001958e"). InnerVolumeSpecName "kube-api-access-wqq4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:25:57.466642 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.466612 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.466642 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.466640 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28c63-7d68-429c-893d-25f7f001958e-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.466642 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.466651 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.466869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.466660 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.466869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.466669 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.466869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.466678 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqq4v\" (UniqueName: \"kubernetes.io/projected/8ba28c63-7d68-429c-893d-25f7f001958e-kube-api-access-wqq4v\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.466869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.466686 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28c63-7d68-429c-893d-25f7f001958e-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.516314 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.516282 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8"] Jul 15 19:25:57.516520 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.516489 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" podUID="4506247d-66eb-4271-86a3-95a33ccfcc40" containerName="maas-api" containerID="cri-o://f5a4c631cfda634aad45ddb8997323eb84a4f0ae5cd01515c8d7b36652d6aebc" gracePeriod=30 Jul 15 19:25:57.559173 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.559118 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:57.686308 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.686276 2582 generic.go:356] "Generic (PLEG): container finished" podID="4506247d-66eb-4271-86a3-95a33ccfcc40" containerID="f5a4c631cfda634aad45ddb8997323eb84a4f0ae5cd01515c8d7b36652d6aebc" exitCode=0 Jul 15 19:25:57.686491 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.686337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" event={"ID":"4506247d-66eb-4271-86a3-95a33ccfcc40","Type":"ContainerDied","Data":"f5a4c631cfda634aad45ddb8997323eb84a4f0ae5cd01515c8d7b36652d6aebc"} Jul 15 19:25:57.687915 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.687887 2582 generic.go:356] "Generic (PLEG): container finished" podID="8ba28c63-7d68-429c-893d-25f7f001958e" containerID="bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d" exitCode=0 Jul 15 19:25:57.688036 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.687928 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" Jul 15 19:25:57.688036 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.687959 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" event={"ID":"8ba28c63-7d68-429c-893d-25f7f001958e","Type":"ContainerDied","Data":"bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d"} Jul 15 19:25:57.688036 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.688018 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb" event={"ID":"8ba28c63-7d68-429c-893d-25f7f001958e","Type":"ContainerDied","Data":"ac51280dd6b709baff76e146796541204b061d77bcf99c950a91b65b94434d8c"} Jul 15 19:25:57.688210 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.688052 2582 scope.go:117] "RemoveContainer" containerID="bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d" Jul 15 19:25:57.701942 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.701921 2582 scope.go:117] "RemoveContainer" containerID="bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d" Jul 15 19:25:57.702283 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:25:57.702258 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d\": container with ID starting with bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d not found: ID does not exist" containerID="bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d" Jul 15 19:25:57.702382 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.702290 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d"} err="failed to get container status \"bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d\": rpc error: code = NotFound desc = could not find container \"bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d\": container with ID starting with bab2e01c99a31b724b27e94db5e5be4a4f2fe9e51df5056f402c2944c4fe6d1d not found: ID does not exist" Jul 15 19:25:57.723739 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.723710 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb"] Jul 15 19:25:57.735414 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.735382 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-62375b76-openshift-default-6f468c85d7-n8njb"] Jul 15 19:25:57.780731 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.780704 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:57.871262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.871209 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls\") pod \"4506247d-66eb-4271-86a3-95a33ccfcc40\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " Jul 15 19:25:57.871443 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.871298 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpsw7\" (UniqueName: \"kubernetes.io/projected/4506247d-66eb-4271-86a3-95a33ccfcc40-kube-api-access-qpsw7\") pod \"4506247d-66eb-4271-86a3-95a33ccfcc40\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " Jul 15 19:25:57.871443 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.871391 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4506247d-66eb-4271-86a3-95a33ccfcc40-openshift-service-ca\") pod \"4506247d-66eb-4271-86a3-95a33ccfcc40\" (UID: \"4506247d-66eb-4271-86a3-95a33ccfcc40\") " Jul 15 19:25:57.871740 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.871715 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4506247d-66eb-4271-86a3-95a33ccfcc40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4506247d-66eb-4271-86a3-95a33ccfcc40" (UID: "4506247d-66eb-4271-86a3-95a33ccfcc40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:25:57.873502 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.873472 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4506247d-66eb-4271-86a3-95a33ccfcc40-kube-api-access-qpsw7" (OuterVolumeSpecName: "kube-api-access-qpsw7") pod "4506247d-66eb-4271-86a3-95a33ccfcc40" (UID: "4506247d-66eb-4271-86a3-95a33ccfcc40"). InnerVolumeSpecName "kube-api-access-qpsw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:25:57.873502 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.873481 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4506247d-66eb-4271-86a3-95a33ccfcc40" (UID: "4506247d-66eb-4271-86a3-95a33ccfcc40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:25:57.973136 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.973108 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4506247d-66eb-4271-86a3-95a33ccfcc40-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.973136 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.973136 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpsw7\" (UniqueName: \"kubernetes.io/projected/4506247d-66eb-4271-86a3-95a33ccfcc40-kube-api-access-qpsw7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:57.973347 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:57.973146 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4506247d-66eb-4271-86a3-95a33ccfcc40-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:25:58.201134 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:58.201105 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba28c63-7d68-429c-893d-25f7f001958e" path="/var/lib/kubelet/pods/8ba28c63-7d68-429c-893d-25f7f001958e/volumes" Jul 15 19:25:58.694185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:58.694149 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" Jul 15 19:25:58.694185 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:58.694169 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8" event={"ID":"4506247d-66eb-4271-86a3-95a33ccfcc40","Type":"ContainerDied","Data":"3ce0b63f194741e323ff8d3597bdf11a6298ec96a902c127351644aaa3a6324b"} Jul 15 19:25:58.694732 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:58.694262 2582 scope.go:117] "RemoveContainer" containerID="f5a4c631cfda634aad45ddb8997323eb84a4f0ae5cd01515c8d7b36652d6aebc" Jul 15 19:25:58.714539 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:58.714516 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8"] Jul 15 19:25:58.721043 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:58.721024 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62375b76-6484464c8f-lgsv8"] Jul 15 19:25:59.040385 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:25:59.040322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:00.199526 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:00.199495 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4506247d-66eb-4271-86a3-95a33ccfcc40" path="/var/lib/kubelet/pods/4506247d-66eb-4271-86a3-95a33ccfcc40/volumes" Jul 15 19:26:05.106176 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.106139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:05.602114 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.602085 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg"] Jul 15 19:26:05.602702 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.602675 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba28c63-7d68-429c-893d-25f7f001958e" containerName="istio-proxy" Jul 15 19:26:05.602702 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.602701 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba28c63-7d68-429c-893d-25f7f001958e" containerName="istio-proxy" Jul 15 19:26:05.602908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.602723 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4506247d-66eb-4271-86a3-95a33ccfcc40" containerName="maas-api" Jul 15 19:26:05.602908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.602733 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4506247d-66eb-4271-86a3-95a33ccfcc40" containerName="maas-api" Jul 15 19:26:05.602908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.602832 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4506247d-66eb-4271-86a3-95a33ccfcc40" containerName="maas-api" Jul 15 19:26:05.602908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.602842 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba28c63-7d68-429c-893d-25f7f001958e" containerName="istio-proxy" Jul 15 19:26:05.605738 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.605717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.608446 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.608427 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bcc2cd9f-gw-openshift-default-dockercfg-wsw4q\"" Jul 15 19:26:05.622427 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.622405 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg"] Jul 15 19:26:05.744161 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-token\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744352 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-socket\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744352 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744208 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-certs\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744352 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-podinfo\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744352 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-envoy\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744513 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-data\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744513 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9m4r\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-kube-api-access-d9m4r\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744513 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744457 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b982eb-3028-4719-bd49-4523f5d37d8c-istiod-ca-cert\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.744513 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.744506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-credential-socket\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845172 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-envoy\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845172 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-data\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9m4r\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-kube-api-access-d9m4r\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b982eb-3028-4719-bd49-4523f5d37d8c-istiod-ca-cert\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-credential-socket\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-token\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845380 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-socket\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-certs\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-podinfo\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845808 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845700 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-data\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845848 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845812 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-credential-socket\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.845932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845879 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-socket\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.846012 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.845997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-certs\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.846097 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.846074 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b982eb-3028-4719-bd49-4523f5d37d8c-istiod-ca-cert\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.847745 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.847720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-envoy\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.847949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.847934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-podinfo\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.857560 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.857498 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-token\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.857818 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.857800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9m4r\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-kube-api-access-d9m4r\") pod \"e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.916157 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.916118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:05.974274 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.974198 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq"] Jul 15 19:26:05.978200 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.977953 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:05.980739 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.980547 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bcc2cd9f-serving-cert\"" Jul 15 19:26:05.993084 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:05.993050 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq"] Jul 15 19:26:06.046893 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.046861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pwz\" (UniqueName: \"kubernetes.io/projected/47210713-781d-4e0f-bd5c-f5bf2175986a-kube-api-access-z6pwz\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.047079 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.046922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47210713-781d-4e0f-bd5c-f5bf2175986a-openshift-service-ca\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.047079 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.047028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.070943 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.070917 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg"] Jul 15 19:26:06.072729 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:26:06.072696 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b982eb_3028_4719_bd49_4523f5d37d8c.slice/crio-2af9271f5ee3bb7b1af6ce9c09b14aa94c283b61d88c3126e19cdd92acac839d WatchSource:0}: Error finding container 2af9271f5ee3bb7b1af6ce9c09b14aa94c283b61d88c3126e19cdd92acac839d: Status 404 returned error can't find the container with id 2af9271f5ee3bb7b1af6ce9c09b14aa94c283b61d88c3126e19cdd92acac839d Jul 15 19:26:06.075617 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.075565 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:26:06.075721 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.075683 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:26:06.075786 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.075752 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:26:06.148415 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.148385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.148764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.148489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pwz\" (UniqueName: \"kubernetes.io/projected/47210713-781d-4e0f-bd5c-f5bf2175986a-kube-api-access-z6pwz\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.148764 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:26:06.148552 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-serving-cert: secret "maas-api-e2e-migrate-bcc2cd9f-serving-cert" not found Jul 15 19:26:06.148764 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:26:06.148622 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls podName:47210713-781d-4e0f-bd5c-f5bf2175986a nodeName:}" failed. No retries permitted until 2026-07-15 19:26:06.648599328 +0000 UTC m=+1519.043955293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls") pod "maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" (UID: "47210713-781d-4e0f-bd5c-f5bf2175986a") : secret "maas-api-e2e-migrate-bcc2cd9f-serving-cert" not found Jul 15 19:26:06.148764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.148557 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47210713-781d-4e0f-bd5c-f5bf2175986a-openshift-service-ca\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.149194 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.149175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47210713-781d-4e0f-bd5c-f5bf2175986a-openshift-service-ca\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.157101 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.157073 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pwz\" (UniqueName: \"kubernetes.io/projected/47210713-781d-4e0f-bd5c-f5bf2175986a-kube-api-access-z6pwz\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.654769 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.654731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.657189 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.657160 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls\") pod \"maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.740229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.740185 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" event={"ID":"70b982eb-3028-4719-bd49-4523f5d37d8c","Type":"ContainerStarted","Data":"0dbdce9b8baabfa1fb4dc2a33fc17c040654e4937ea8c6cfea17add578fe04cd"} Jul 15 19:26:06.740387 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.740264 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" event={"ID":"70b982eb-3028-4719-bd49-4523f5d37d8c","Type":"ContainerStarted","Data":"2af9271f5ee3bb7b1af6ce9c09b14aa94c283b61d88c3126e19cdd92acac839d"} Jul 15 19:26:06.764227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.764179 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" podStartSLOduration=1.764164188 podStartE2EDuration="1.764164188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:26:06.761262291 +0000 UTC m=+1519.156618258" watchObservedRunningTime="2026-07-15 19:26:06.764164188 +0000 UTC m=+1519.159520158" Jul 15 19:26:06.898007 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.897972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:06.917212 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:06.917189 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:07.034746 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:07.034721 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq"] Jul 15 19:26:07.036334 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:26:07.036298 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47210713_781d_4e0f_bd5c_f5bf2175986a.slice/crio-29fe468a1c51507dbdae44c1f22951f01d00065cc365055bc6ffb1295fd9d540 WatchSource:0}: Error finding container 29fe468a1c51507dbdae44c1f22951f01d00065cc365055bc6ffb1295fd9d540: Status 404 returned error can't find the container with id 29fe468a1c51507dbdae44c1f22951f01d00065cc365055bc6ffb1295fd9d540 Jul 15 19:26:07.149152 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:07.149126 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:07.748038 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:07.747935 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" event={"ID":"47210713-781d-4e0f-bd5c-f5bf2175986a","Type":"ContainerStarted","Data":"51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1"} Jul 15 19:26:07.748038 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:07.747993 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" event={"ID":"47210713-781d-4e0f-bd5c-f5bf2175986a","Type":"ContainerStarted","Data":"29fe468a1c51507dbdae44c1f22951f01d00065cc365055bc6ffb1295fd9d540"} Jul 15 19:26:07.748316 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:07.748177 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:07.778256 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:07.778195 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" podStartSLOduration=2.342830782 podStartE2EDuration="2.778179952s" totalImagesPullingTime="435.34917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:26:07.776027779 +0000 UTC m=+1520.171383762" watchObservedRunningTime="2026-07-15 19:26:07.778179952 +0000 UTC m=+1520.173535957" Jul 15 19:26:07.921441 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:07.921405 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:08.753449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:08.753413 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:08.754203 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:08.754186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:09.600050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:09.600014 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:11.549495 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:11.549408 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg"] Jul 15 19:26:11.593579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:11.593542 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:11.694842 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:11.694785 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:11.768953 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:11.768868 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" podUID="70b982eb-3028-4719-bd49-4523f5d37d8c" containerName="istio-proxy" containerID="cri-o://0dbdce9b8baabfa1fb4dc2a33fc17c040654e4937ea8c6cfea17add578fe04cd" gracePeriod=30 Jul 15 19:26:13.358459 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:13.358428 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:13.759272 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:13.759228 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:17.805437 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.805407 2582 generic.go:356] "Generic (PLEG): container finished" podID="70b982eb-3028-4719-bd49-4523f5d37d8c" containerID="0dbdce9b8baabfa1fb4dc2a33fc17c040654e4937ea8c6cfea17add578fe04cd" exitCode=0 Jul 15 19:26:17.805801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.805476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" event={"ID":"70b982eb-3028-4719-bd49-4523f5d37d8c","Type":"ContainerDied","Data":"0dbdce9b8baabfa1fb4dc2a33fc17c040654e4937ea8c6cfea17add578fe04cd"} Jul 15 19:26:17.821992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.821968 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:17.960305 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960175 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-certs\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960305 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960227 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-credential-socket\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960334 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-token\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960377 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-envoy\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960411 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9m4r\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-kube-api-access-d9m4r\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960482 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:17.960556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960510 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:17.960822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960572 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-data\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960613 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b982eb-3028-4719-bd49-4523f5d37d8c-istiod-ca-cert\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-socket\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.960822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.960696 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-podinfo\") pod \"70b982eb-3028-4719-bd49-4523f5d37d8c\" (UID: \"70b982eb-3028-4719-bd49-4523f5d37d8c\") " Jul 15 19:26:17.961053 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.961020 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:17.961053 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.961040 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:17.961053 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.961040 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b982eb-3028-4719-bd49-4523f5d37d8c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:26:17.961267 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.961219 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:17.961510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.961488 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-data" (OuterVolumeSpecName: "istio-data") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:17.962564 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.962534 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:17.962772 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.962756 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-kube-api-access-d9m4r" (OuterVolumeSpecName: "kube-api-access-d9m4r") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "kube-api-access-d9m4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:26:17.962904 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.962884 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:26:17.962950 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:17.962884 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70b982eb-3028-4719-bd49-4523f5d37d8c" (UID: "70b982eb-3028-4719-bd49-4523f5d37d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:26:18.062428 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.062381 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.062428 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.062422 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9m4r\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-kube-api-access-d9m4r\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.062428 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.062433 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.062428 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.062441 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b982eb-3028-4719-bd49-4523f5d37d8c-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.062428 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.062449 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b982eb-3028-4719-bd49-4523f5d37d8c-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.062780 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.062457 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.062780 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.062465 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b982eb-3028-4719-bd49-4523f5d37d8c-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.104135 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.104098 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq"] Jul 15 19:26:18.104512 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.104341 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" podUID="47210713-781d-4e0f-bd5c-f5bf2175986a" containerName="maas-api" containerID="cri-o://51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1" gracePeriod=30 Jul 15 19:26:18.145830 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.145801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:18.326517 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.326473 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:18.350628 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.350603 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:18.466666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.466636 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6pwz\" (UniqueName: \"kubernetes.io/projected/47210713-781d-4e0f-bd5c-f5bf2175986a-kube-api-access-z6pwz\") pod \"47210713-781d-4e0f-bd5c-f5bf2175986a\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " Jul 15 19:26:18.466886 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.466749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47210713-781d-4e0f-bd5c-f5bf2175986a-openshift-service-ca\") pod \"47210713-781d-4e0f-bd5c-f5bf2175986a\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " Jul 15 19:26:18.466886 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.466783 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls\") pod \"47210713-781d-4e0f-bd5c-f5bf2175986a\" (UID: \"47210713-781d-4e0f-bd5c-f5bf2175986a\") " Jul 15 19:26:18.467111 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.467087 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47210713-781d-4e0f-bd5c-f5bf2175986a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47210713-781d-4e0f-bd5c-f5bf2175986a" (UID: "47210713-781d-4e0f-bd5c-f5bf2175986a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:26:18.468997 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.468961 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47210713-781d-4e0f-bd5c-f5bf2175986a-kube-api-access-z6pwz" (OuterVolumeSpecName: "kube-api-access-z6pwz") pod "47210713-781d-4e0f-bd5c-f5bf2175986a" (UID: "47210713-781d-4e0f-bd5c-f5bf2175986a"). InnerVolumeSpecName "kube-api-access-z6pwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:26:18.469115 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.468994 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47210713-781d-4e0f-bd5c-f5bf2175986a" (UID: "47210713-781d-4e0f-bd5c-f5bf2175986a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:26:18.567750 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.567666 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6pwz\" (UniqueName: \"kubernetes.io/projected/47210713-781d-4e0f-bd5c-f5bf2175986a-kube-api-access-z6pwz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.567750 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.567692 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47210713-781d-4e0f-bd5c-f5bf2175986a-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.567750 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.567701 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47210713-781d-4e0f-bd5c-f5bf2175986a-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:18.812251 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.812222 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" Jul 15 19:26:18.812692 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.812270 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg" event={"ID":"70b982eb-3028-4719-bd49-4523f5d37d8c","Type":"ContainerDied","Data":"2af9271f5ee3bb7b1af6ce9c09b14aa94c283b61d88c3126e19cdd92acac839d"} Jul 15 19:26:18.812692 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.812333 2582 scope.go:117] "RemoveContainer" containerID="0dbdce9b8baabfa1fb4dc2a33fc17c040654e4937ea8c6cfea17add578fe04cd" Jul 15 19:26:18.813868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.813646 2582 generic.go:356] "Generic (PLEG): container finished" podID="47210713-781d-4e0f-bd5c-f5bf2175986a" containerID="51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1" exitCode=0 Jul 15 19:26:18.813868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.813693 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" Jul 15 19:26:18.813868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.813698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" event={"ID":"47210713-781d-4e0f-bd5c-f5bf2175986a","Type":"ContainerDied","Data":"51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1"} Jul 15 19:26:18.813868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.813815 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq" event={"ID":"47210713-781d-4e0f-bd5c-f5bf2175986a","Type":"ContainerDied","Data":"29fe468a1c51507dbdae44c1f22951f01d00065cc365055bc6ffb1295fd9d540"} Jul 15 19:26:18.821615 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.821593 2582 scope.go:117] "RemoveContainer" containerID="51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1" Jul 15 19:26:18.829655 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.829640 2582 scope.go:117] "RemoveContainer" containerID="51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1" Jul 15 19:26:18.829901 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:26:18.829883 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1\": container with ID starting with 51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1 not found: ID does not exist" containerID="51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1" Jul 15 19:26:18.829949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.829909 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1"} err="failed to get container status \"51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1\": rpc error: code = NotFound desc = could not find container \"51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1\": container with ID starting with 51a82f3cc159e711429a517ee8f3def6337dfbb912c7de692bb259a7a9398de1 not found: ID does not exist" Jul 15 19:26:18.839517 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.839495 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg"] Jul 15 19:26:18.846316 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.846299 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bcc2cd9f-gw-openshift-default-586d9dd6bc-cqjcg"] Jul 15 19:26:18.868028 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.868003 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq"] Jul 15 19:26:18.872499 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:18.872477 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcc2cd9f-cd545bddb-fjrvq"] Jul 15 19:26:20.199902 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:20.199871 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47210713-781d-4e0f-bd5c-f5bf2175986a" path="/var/lib/kubelet/pods/47210713-781d-4e0f-bd5c-f5bf2175986a/volumes" Jul 15 19:26:20.200317 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:20.200304 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b982eb-3028-4719-bd49-4523f5d37d8c" path="/var/lib/kubelet/pods/70b982eb-3028-4719-bd49-4523f5d37d8c/volumes" Jul 15 19:26:24.383719 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.383682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:24.593992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.593958 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:24.650911 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.650824 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw"] Jul 15 19:26:24.651308 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.651290 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47210713-781d-4e0f-bd5c-f5bf2175986a" containerName="maas-api" Jul 15 19:26:24.651408 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.651310 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="47210713-781d-4e0f-bd5c-f5bf2175986a" containerName="maas-api" Jul 15 19:26:24.651408 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.651326 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b982eb-3028-4719-bd49-4523f5d37d8c" containerName="istio-proxy" Jul 15 19:26:24.651408 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.651334 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b982eb-3028-4719-bd49-4523f5d37d8c" containerName="istio-proxy" Jul 15 19:26:24.651561 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.651425 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b982eb-3028-4719-bd49-4523f5d37d8c" containerName="istio-proxy" Jul 15 19:26:24.651561 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.651440 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="47210713-781d-4e0f-bd5c-f5bf2175986a" containerName="maas-api" Jul 15 19:26:24.656084 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.656053 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.659670 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.659650 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-38656213-openshift-default-dockercfg-7p2rs\"" Jul 15 19:26:24.680032 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.680009 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw"] Jul 15 19:26:24.723260 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723213 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-certs\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723260 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-istio-token\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723419 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642aab4d-e213-4d4e-96a9-a06a84003060-istiod-ca-cert\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723419 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723378 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-socket\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723500 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bdl\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-kube-api-access-w6bdl\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723532 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-envoy\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723566 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642aab4d-e213-4d4e-96a9-a06a84003060-istio-podinfo\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723603 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-data\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.723637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.723623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-credential-socket\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.732490 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.732454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:24.824029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.823992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-envoy\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824032 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642aab4d-e213-4d4e-96a9-a06a84003060-istio-podinfo\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-data\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-credential-socket\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-certs\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-istio-token\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642aab4d-e213-4d4e-96a9-a06a84003060-istiod-ca-cert\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824274 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-socket\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bdl\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-kube-api-access-w6bdl\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824711 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824592 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-data\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824711 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-credential-socket\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824711 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-socket\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.824905 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.824883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-certs\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.825166 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.825139 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642aab4d-e213-4d4e-96a9-a06a84003060-istiod-ca-cert\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.826484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.826461 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-envoy\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.826869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.826849 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642aab4d-e213-4d4e-96a9-a06a84003060-istio-podinfo\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.837318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.837287 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bdl\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-kube-api-access-w6bdl\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.838034 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.838011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-istio-token\") pod \"e2e-derive-38656213-openshift-default-ff858b687-jfthw\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:24.927613 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.927531 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:24.966984 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:24.966952 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:25.089479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.089445 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:25.143487 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.143454 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw"] Jul 15 19:26:25.146768 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:26:25.146738 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642aab4d_e213_4d4e_96a9_a06a84003060.slice/crio-05c85435f2db5406c1c379c70d192e92502b003e4a8b61a7a692abb43c47045a WatchSource:0}: Error finding container 05c85435f2db5406c1c379c70d192e92502b003e4a8b61a7a692abb43c47045a: Status 404 returned error can't find the container with id 05c85435f2db5406c1c379c70d192e92502b003e4a8b61a7a692abb43c47045a Jul 15 19:26:25.148717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.148684 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:26:25.148825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.148759 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:26:25.148825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.148805 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:26:25.599602 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.599568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:25.855091 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.855013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" event={"ID":"642aab4d-e213-4d4e-96a9-a06a84003060","Type":"ContainerStarted","Data":"213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2"} Jul 15 19:26:25.855091 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.855088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" event={"ID":"642aab4d-e213-4d4e-96a9-a06a84003060","Type":"ContainerStarted","Data":"05c85435f2db5406c1c379c70d192e92502b003e4a8b61a7a692abb43c47045a"} Jul 15 19:26:25.967678 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:25.967645 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:26.099169 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:26.099134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:26.406051 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:26.406022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:26.972223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:26.972194 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:27.014313 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:27.014259 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" podStartSLOduration=3.014232247 podStartE2EDuration="3.014232247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:26:25.900450995 +0000 UTC m=+1538.295806965" watchObservedRunningTime="2026-07-15 19:26:27.014232247 +0000 UTC m=+1539.409588218" Jul 15 19:26:27.865372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:27.865334 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:27.866428 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:27.866410 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:28.233699 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:28.233667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.296681 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:30.296646 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw"] Jul 15 19:26:30.334568 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:30.334531 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.437216 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:30.437188 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.881513 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:30.881478 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" podUID="642aab4d-e213-4d4e-96a9-a06a84003060" containerName="istio-proxy" containerID="cri-o://213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2" gracePeriod=30 Jul 15 19:26:32.626886 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:32.626852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:36.144443 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.144392 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:36.231702 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231667 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6bdl\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-kube-api-access-w6bdl\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.231702 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231704 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-credential-socket\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.231932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231771 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642aab4d-e213-4d4e-96a9-a06a84003060-istiod-ca-cert\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.231932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231810 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-istio-token\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.231932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231863 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-envoy\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.231932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231905 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-certs\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.232139 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642aab4d-e213-4d4e-96a9-a06a84003060-istio-podinfo\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.232139 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.231970 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-data\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.232139 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.232000 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-socket\") pod \"642aab4d-e213-4d4e-96a9-a06a84003060\" (UID: \"642aab4d-e213-4d4e-96a9-a06a84003060\") " Jul 15 19:26:36.232139 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.232044 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:36.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.232203 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642aab4d-e213-4d4e-96a9-a06a84003060-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:26:36.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.232299 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.232396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.232372 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:36.232556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.232433 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-data" (OuterVolumeSpecName: "istio-data") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:36.232556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.232516 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:36.233999 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.233958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-istio-token" (OuterVolumeSpecName: "istio-token") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:26:36.234122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.234037 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-kube-api-access-w6bdl" (OuterVolumeSpecName: "kube-api-access-w6bdl") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "kube-api-access-w6bdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:26:36.234122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.234044 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:26:36.234122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.234105 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/642aab4d-e213-4d4e-96a9-a06a84003060-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "642aab4d-e213-4d4e-96a9-a06a84003060" (UID: "642aab4d-e213-4d4e-96a9-a06a84003060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:26:36.305815 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.305759 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:36.333681 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333649 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642aab4d-e213-4d4e-96a9-a06a84003060-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.333681 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333676 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.333878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333689 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.333878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333700 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.333878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333712 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642aab4d-e213-4d4e-96a9-a06a84003060-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.333878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333723 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.333878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333735 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642aab4d-e213-4d4e-96a9-a06a84003060-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.333878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.333748 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bdl\" (UniqueName: \"kubernetes.io/projected/642aab4d-e213-4d4e-96a9-a06a84003060-kube-api-access-w6bdl\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:26:36.915823 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.915784 2582 generic.go:356] "Generic (PLEG): container finished" podID="642aab4d-e213-4d4e-96a9-a06a84003060" containerID="213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2" exitCode=0 Jul 15 19:26:36.915995 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.915848 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" Jul 15 19:26:36.915995 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.915862 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" event={"ID":"642aab4d-e213-4d4e-96a9-a06a84003060","Type":"ContainerDied","Data":"213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2"} Jul 15 19:26:36.915995 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.915941 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw" event={"ID":"642aab4d-e213-4d4e-96a9-a06a84003060","Type":"ContainerDied","Data":"05c85435f2db5406c1c379c70d192e92502b003e4a8b61a7a692abb43c47045a"} Jul 15 19:26:36.915995 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.915964 2582 scope.go:117] "RemoveContainer" containerID="213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2" Jul 15 19:26:36.925976 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.925954 2582 scope.go:117] "RemoveContainer" containerID="213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2" Jul 15 19:26:36.926210 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:26:36.926191 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2\": container with ID starting with 213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2 not found: ID does not exist" containerID="213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2" Jul 15 19:26:36.926304 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.926218 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2"} err="failed to get container status \"213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2\": rpc error: code = NotFound desc = could not find container \"213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2\": container with ID starting with 213827ef10ac062a4959ff47318a6963a79d7b128e0c278e75f6694ff82a94b2 not found: ID does not exist" Jul 15 19:26:36.943396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.943373 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw"] Jul 15 19:26:36.947899 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.947875 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-38656213-openshift-default-ff858b687-jfthw"] Jul 15 19:26:36.967910 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:36.967887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:38.199624 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:38.199592 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="642aab4d-e213-4d4e-96a9-a06a84003060" path="/var/lib/kubelet/pods/642aab4d-e213-4d4e-96a9-a06a84003060/volumes" Jul 15 19:26:43.611917 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:43.611886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:43.716416 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:43.716381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:44.180991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:44.180953 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:44.329310 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:44.329274 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:46.056350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:46.056316 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:52.099365 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:52.099332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:52.242142 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:52.242107 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:52.377186 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:52.377109 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:52.465444 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:52.465413 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:52.846851 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:26:52.846814 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:18.794331 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:18.794296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:24.585710 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:24.585677 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:24.691210 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:24.691177 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:24.788008 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:24.787972 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:37.283787 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:37.283755 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:40.704578 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:40.704491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:46.544914 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:46.544884 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:46.614813 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:46.614788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:46.716942 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:46.716895 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:48.436936 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:48.436905 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:57.236334 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:57.236298 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:58.105201 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:27:58.105167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:04.139651 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:04.139613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:04.246032 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:04.246001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:04.368504 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:04.368472 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:04.477517 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:04.477490 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:05.378644 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:05.378614 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:05.473527 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:05.473499 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:07.197047 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:07.197015 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:13.174540 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:13.174507 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:13.279481 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:13.279447 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:13.808508 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:13.808470 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:19.810036 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:19.810006 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:19.897642 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:19.897613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:20.040719 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.040688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:20.341108 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.341074 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd"] Jul 15 19:28:20.341473 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.341460 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="642aab4d-e213-4d4e-96a9-a06a84003060" containerName="istio-proxy" Jul 15 19:28:20.341529 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.341475 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="642aab4d-e213-4d4e-96a9-a06a84003060" containerName="istio-proxy" Jul 15 19:28:20.341575 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.341558 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="642aab4d-e213-4d4e-96a9-a06a84003060" containerName="istio-proxy" Jul 15 19:28:20.344798 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.344776 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.347404 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.347385 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e84500d1-openshift-default-dockercfg-9v7pg\"" Jul 15 19:28:20.369289 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.369263 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd"] Jul 15 19:28:20.373822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.373798 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:20.446030 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-data\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-credential-socket\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6h9r\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-kube-api-access-s6h9r\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-certs\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-istio-token\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446229 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446221 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c06c5a1-0175-4022-bf63-887c60c57938-istiod-ca-cert\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446527 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446387 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-envoy\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446527 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-socket\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.446527 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.446511 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c06c5a1-0175-4022-bf63-887c60c57938-istio-podinfo\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-socket\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c06c5a1-0175-4022-bf63-887c60c57938-istio-podinfo\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-data\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-credential-socket\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6h9r\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-kube-api-access-s6h9r\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547217 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-certs\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-istio-token\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c06c5a1-0175-4022-bf63-887c60c57938-istiod-ca-cert\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547354 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-envoy\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547788 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-socket\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547788 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547645 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-data\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547788 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-credential-socket\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.547993 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.547966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-certs\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.548207 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.548186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c06c5a1-0175-4022-bf63-887c60c57938-istiod-ca-cert\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.549616 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.549597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-envoy\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.549750 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.549732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c06c5a1-0175-4022-bf63-887c60c57938-istio-podinfo\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.556966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.556946 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-istio-token\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.557177 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.557153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6h9r\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-kube-api-access-s6h9r\") pod \"e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.589490 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.589463 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:20.654700 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.654629 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:20.798992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.798969 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd"] Jul 15 19:28:20.801085 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:28:20.801052 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c06c5a1_0175_4022_bf63_887c60c57938.slice/crio-d93006a072c190331ad10e0fbf692aa27c29369bfae178d99537270bfd449ff6 WatchSource:0}: Error finding container d93006a072c190331ad10e0fbf692aa27c29369bfae178d99537270bfd449ff6: Status 404 returned error can't find the container with id d93006a072c190331ad10e0fbf692aa27c29369bfae178d99537270bfd449ff6 Jul 15 19:28:20.803250 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.803205 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:28:20.803342 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.803288 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:28:20.803342 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:20.803321 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:28:21.434730 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:21.434696 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" event={"ID":"1c06c5a1-0175-4022-bf63-887c60c57938","Type":"ContainerStarted","Data":"1883f2875c4e1dff357c27d695b966875bcdd03c2b5ea97a05214a96dbbd4ac5"} Jul 15 19:28:21.435152 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:21.434746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" event={"ID":"1c06c5a1-0175-4022-bf63-887c60c57938","Type":"ContainerStarted","Data":"d93006a072c190331ad10e0fbf692aa27c29369bfae178d99537270bfd449ff6"} Jul 15 19:28:21.456282 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:21.456218 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" podStartSLOduration=1.456204232 podStartE2EDuration="1.456204232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:28:21.455692164 +0000 UTC m=+1653.851048149" watchObservedRunningTime="2026-07-15 19:28:21.456204232 +0000 UTC m=+1653.851560278" Jul 15 19:28:21.655487 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:21.655452 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:21.659900 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:21.659877 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:22.372382 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:22.372345 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:22.440100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:22.440063 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:22.441016 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:22.440999 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:22.632973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:22.632894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:22.730400 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:22.730374 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:23.104558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:23.104525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:23.205031 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:23.205003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:23.305128 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:23.305086 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:27.898630 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:27.898586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:28.572068 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:28.572022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:28.781566 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:28.781511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:29.006862 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:29.006829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:34.750605 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:34.750577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:34.816577 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:34.816547 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:35.503014 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:35.502970 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:37.441821 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.441785 2582 patch_prober.go:28] interesting pod/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 15 19:28:37.442228 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.441847 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 15 19:28:37.527081 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.527050 2582 generic.go:356] "Generic (PLEG): container finished" podID="1c06c5a1-0175-4022-bf63-887c60c57938" containerID="1883f2875c4e1dff357c27d695b966875bcdd03c2b5ea97a05214a96dbbd4ac5" exitCode=0 Jul 15 19:28:37.527263 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.527109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" event={"ID":"1c06c5a1-0175-4022-bf63-887c60c57938","Type":"ContainerDied","Data":"1883f2875c4e1dff357c27d695b966875bcdd03c2b5ea97a05214a96dbbd4ac5"} Jul 15 19:28:37.527481 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.527468 2582 scope.go:117] "RemoveContainer" containerID="1883f2875c4e1dff357c27d695b966875bcdd03c2b5ea97a05214a96dbbd4ac5" Jul 15 19:28:37.528459 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.528429 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:28:37.528523 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.528505 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:28:37.528565 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:37.528537 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:28:38.533165 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:38.533132 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" event={"ID":"1c06c5a1-0175-4022-bf63-887c60c57938","Type":"ContainerStarted","Data":"dcb7f38159c57e00b550ea08beca88d93952ceee780ce7ccdc52721a3cb345d7"} Jul 15 19:28:38.654763 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:38.654723 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:38.655080 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:38.655058 2582 patch_prober.go:28] interesting pod/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 15 19:28:38.655134 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:38.655107 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 15 19:28:39.655317 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:39.655280 2582 patch_prober.go:28] interesting pod/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 15 19:28:39.655710 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:39.655355 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 15 19:28:40.623259 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:40.623216 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:40.655628 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:40.655601 2582 patch_prober.go:28] interesting pod/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 15 19:28:40.655999 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:40.655652 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 15 19:28:41.629729 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:41.629693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:41.655050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:41.655014 2582 patch_prober.go:28] interesting pod/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 15 19:28:41.655211 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:41.655068 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 15 19:28:42.659861 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:42.659830 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:42.660328 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:42.660092 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:42.660919 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:42.660895 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:43.867962 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:43.867931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:43.968843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:43.968802 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:47.200520 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.200485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:47.300349 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.300316 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:47.456313 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.456215 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd"] Jul 15 19:28:47.456456 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.456414 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" containerID="cri-o://dcb7f38159c57e00b550ea08beca88d93952ceee780ce7ccdc52721a3cb345d7" gracePeriod=30 Jul 15 19:28:47.485260 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.485199 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:47.737986 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.737952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:47.844738 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.844703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:47.947630 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:47.947592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:48.082786 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:48.082696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:52.617811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.617779 2582 generic.go:356] "Generic (PLEG): container finished" podID="1c06c5a1-0175-4022-bf63-887c60c57938" containerID="dcb7f38159c57e00b550ea08beca88d93952ceee780ce7ccdc52721a3cb345d7" exitCode=0 Jul 15 19:28:52.618273 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.617853 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" event={"ID":"1c06c5a1-0175-4022-bf63-887c60c57938","Type":"ContainerDied","Data":"dcb7f38159c57e00b550ea08beca88d93952ceee780ce7ccdc52721a3cb345d7"} Jul 15 19:28:52.618273 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.617916 2582 scope.go:117] "RemoveContainer" containerID="1883f2875c4e1dff357c27d695b966875bcdd03c2b5ea97a05214a96dbbd4ac5" Jul 15 19:28:52.720577 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.720544 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:52.831966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.831871 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-credential-socket\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.831966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.831912 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-certs\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.831966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.831945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6h9r\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-kube-api-access-s6h9r\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.831974 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-envoy\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.831991 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c06c5a1-0175-4022-bf63-887c60c57938-istio-podinfo\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832033 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-data\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832051 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-socket\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832093 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-istio-token\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832121 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c06c5a1-0175-4022-bf63-887c60c57938-istiod-ca-cert\") pod \"1c06c5a1-0175-4022-bf63-887c60c57938\" (UID: \"1c06c5a1-0175-4022-bf63-887c60c57938\") " Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832178 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:28:52.832298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832226 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:28:52.832685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832389 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:28:52.832685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832550 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.832685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832571 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.832685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832587 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.832685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.832602 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c06c5a1-0175-4022-bf63-887c60c57938-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:28:52.833457 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.833428 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:28:52.834499 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.834473 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c06c5a1-0175-4022-bf63-887c60c57938-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:28:52.834499 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.834477 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:28:52.834648 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.834501 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-kube-api-access-s6h9r" (OuterVolumeSpecName: "kube-api-access-s6h9r") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "kube-api-access-s6h9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:28:52.834648 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.834620 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c06c5a1-0175-4022-bf63-887c60c57938" (UID: "1c06c5a1-0175-4022-bf63-887c60c57938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:28:52.933683 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.933647 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6h9r\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-kube-api-access-s6h9r\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.933858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.933700 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.933858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.933710 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c06c5a1-0175-4022-bf63-887c60c57938-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.933858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.933718 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c06c5a1-0175-4022-bf63-887c60c57938-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.933858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.933726 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c06c5a1-0175-4022-bf63-887c60c57938-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:52.933858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:52.933733 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c06c5a1-0175-4022-bf63-887c60c57938-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:28:53.624715 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:53.624687 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" Jul 15 19:28:53.625132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:53.624727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd" event={"ID":"1c06c5a1-0175-4022-bf63-887c60c57938","Type":"ContainerDied","Data":"d93006a072c190331ad10e0fbf692aa27c29369bfae178d99537270bfd449ff6"} Jul 15 19:28:53.625132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:53.624789 2582 scope.go:117] "RemoveContainer" containerID="dcb7f38159c57e00b550ea08beca88d93952ceee780ce7ccdc52721a3cb345d7" Jul 15 19:28:53.646607 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:53.646580 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd"] Jul 15 19:28:53.651029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:53.651003 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e84500d1-openshift-default-6b845d48c9-hckkd"] Jul 15 19:28:54.029949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:54.029916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:54.103684 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:54.103627 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:54.180977 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:54.180947 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:54.201465 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:54.201430 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" path="/var/lib/kubelet/pods/1c06c5a1-0175-4022-bf63-887c60c57938/volumes" Jul 15 19:28:54.284200 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:54.284117 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:54.425904 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:28:54.425873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:00.319334 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:00.319300 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:00.386085 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:00.386051 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:07.899607 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:07.899575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:07.999457 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:07.999420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:08.099485 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:08.099447 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:08.767366 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:08.767329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:08.876362 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:08.876329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:08.962570 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:08.962536 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:10.315052 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:10.314964 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:16.274774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:16.274723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:16.354856 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:16.354821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:16.955824 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:16.955788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:22.743544 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:22.743507 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:22.871825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:22.871792 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:22.964257 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:22.964214 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:33.453906 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:33.453869 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:33.803092 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:33.803050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:54.730469 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:29:54.730420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:00.860119 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:00.860086 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:00.960127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:00.960090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:01.179688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.179603 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:01.536443 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.536412 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp"] Jul 15 19:30:01.536784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.536772 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" Jul 15 19:30:01.536843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.536785 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" Jul 15 19:30:01.536882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.536856 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" Jul 15 19:30:01.536882 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.536865 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" Jul 15 19:30:01.539914 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.539898 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.542759 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.542739 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4256e5-openshift-default-dockercfg-2mrlv\"" Jul 15 19:30:01.577924 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.577889 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp"] Jul 15 19:30:01.581235 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.581208 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:01.651385 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d484ae-d2af-482b-8622-9a61f4d96461-istio-podinfo\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651385 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-istio-token\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651597 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-certs\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651597 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-credential-socket\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651597 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-socket\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651597 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-data\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651597 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdq72\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-kube-api-access-fdq72\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651762 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-envoy\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.651762 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.651652 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d484ae-d2af-482b-8622-9a61f4d96461-istiod-ca-cert\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.743989 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.743956 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:01.752373 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-data\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752530 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752382 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdq72\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-kube-api-access-fdq72\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752530 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-envoy\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752530 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752444 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d484ae-d2af-482b-8622-9a61f4d96461-istiod-ca-cert\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752530 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d484ae-d2af-482b-8622-9a61f4d96461-istio-podinfo\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752530 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-istio-token\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752816 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-certs\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752816 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-credential-socket\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752816 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-socket\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.752816 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.752811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-data\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.753129 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.753070 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-certs\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.753211 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.753130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-credential-socket\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.753211 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.753148 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-socket\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.753351 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.753306 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d484ae-d2af-482b-8622-9a61f4d96461-istiod-ca-cert\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.754731 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.754712 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-envoy\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.754963 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.754945 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d484ae-d2af-482b-8622-9a61f4d96461-istio-podinfo\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.761439 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.761419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdq72\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-kube-api-access-fdq72\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.764685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.764660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-istio-token\") pod \"e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.852386 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.852288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:01.996095 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.996074 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp"] Jul 15 19:30:01.997901 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:30:01.997879 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d484ae_d2af_482b_8622_9a61f4d96461.slice/crio-1fa6be596cf00efaa8ae7e7955003f48c0ad2fc75851a913fa3d79c9e380a8b1 WatchSource:0}: Error finding container 1fa6be596cf00efaa8ae7e7955003f48c0ad2fc75851a913fa3d79c9e380a8b1: Status 404 returned error can't find the container with id 1fa6be596cf00efaa8ae7e7955003f48c0ad2fc75851a913fa3d79c9e380a8b1 Jul 15 19:30:01.999884 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.999854 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:01.999946 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.999920 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:01.999985 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:01.999951 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:03.000571 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:03.000529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" event={"ID":"90d484ae-d2af-482b-8622-9a61f4d96461","Type":"ContainerStarted","Data":"39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9"} Jul 15 19:30:03.000981 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:03.000613 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" event={"ID":"90d484ae-d2af-482b-8622-9a61f4d96461","Type":"ContainerStarted","Data":"1fa6be596cf00efaa8ae7e7955003f48c0ad2fc75851a913fa3d79c9e380a8b1"} Jul 15 19:30:03.024195 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:03.024135 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" podStartSLOduration=2.02411917 podStartE2EDuration="2.02411917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:03.021704058 +0000 UTC m=+1755.417060041" watchObservedRunningTime="2026-07-15 19:30:03.02411917 +0000 UTC m=+1755.419475199" Jul 15 19:30:03.852913 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:03.852881 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:03.858293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:03.858269 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:03.913958 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:03.913919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:04.006046 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:04.006013 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:04.006876 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:04.006859 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:04.014510 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:04.014479 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:04.112403 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:04.112315 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:04.318178 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:04.318141 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:04.712733 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:04.712696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:04.815484 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:04.815398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:09.406166 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:09.406133 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:09.733773 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:09.733722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:10.002426 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.002347 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf"] Jul 15 19:30:10.002743 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.002730 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" Jul 15 19:30:10.002812 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.002744 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06c5a1-0175-4022-bf63-887c60c57938" containerName="istio-proxy" Jul 15 19:30:10.006082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.006063 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.009962 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.009942 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-381b50-openshift-default-dockercfg-l8r6f\"" Jul 15 19:30:10.018301 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.018275 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf"] Jul 15 19:30:10.037884 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.037845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:10.124635 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.124605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-token\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.124825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.124643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-envoy\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.124825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.124660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-podinfo\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.124825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.124808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xhwh\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-kube-api-access-2xhwh\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.124959 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.124861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-data\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.124959 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.124931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-certs\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.125082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.124956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f532e1c9-22b8-406d-97c4-d621762a8cbe-istiod-ca-cert\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.125082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.125002 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-credential-socket\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.125082 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.125044 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-socket\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.225526 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xhwh\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-kube-api-access-2xhwh\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.225526 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-data\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.225793 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-certs\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.225793 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f532e1c9-22b8-406d-97c4-d621762a8cbe-istiod-ca-cert\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.225793 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-credential-socket\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.225793 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225781 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-socket\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226015 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-token\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226015 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-envoy\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226015 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-podinfo\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226015 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.225978 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-data\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226015 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.226004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-certs\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226200 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.226160 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-credential-socket\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226277 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.226257 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-socket\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.226464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.226440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f532e1c9-22b8-406d-97c4-d621762a8cbe-istiod-ca-cert\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.228339 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.228320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-envoy\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.228451 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.228435 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-podinfo\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.234486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.234466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xhwh\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-kube-api-access-2xhwh\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.236881 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.236860 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-token\") pod \"e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.261358 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.261300 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:10.318296 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.318236 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:10.484146 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.484118 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf"] Jul 15 19:30:10.484586 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:30:10.484565 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf532e1c9_22b8_406d_97c4_d621762a8cbe.slice/crio-e71137b345565dbca51693bc39a797a04749261fd8fba06ce7d80ca80afc66f3 WatchSource:0}: Error finding container e71137b345565dbca51693bc39a797a04749261fd8fba06ce7d80ca80afc66f3: Status 404 returned error can't find the container with id e71137b345565dbca51693bc39a797a04749261fd8fba06ce7d80ca80afc66f3 Jul 15 19:30:10.486695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.486662 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:10.486787 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.486726 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:10.486787 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:10.486751 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:11.045814 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:11.045782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" event={"ID":"f532e1c9-22b8-406d-97c4-d621762a8cbe","Type":"ContainerStarted","Data":"9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2"} Jul 15 19:30:11.045992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:11.045832 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" event={"ID":"f532e1c9-22b8-406d-97c4-d621762a8cbe","Type":"ContainerStarted","Data":"e71137b345565dbca51693bc39a797a04749261fd8fba06ce7d80ca80afc66f3"} Jul 15 19:30:11.066143 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:11.066095 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" podStartSLOduration=2.06607869 podStartE2EDuration="2.06607869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:11.064340977 +0000 UTC m=+1763.459696964" watchObservedRunningTime="2026-07-15 19:30:11.06607869 +0000 UTC m=+1763.461434659" Jul 15 19:30:11.318972 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:11.318900 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:11.324050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:11.324025 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:11.933292 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:11.933226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:12.035786 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:12.035753 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:12.050556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:12.050533 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:12.051550 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:12.051531 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:12.199666 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:12.199582 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:12.375565 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:12.375538 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:12.889051 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:12.889017 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:12.995336 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:12.995303 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:16.896584 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:16.896543 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:16.996549 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:16.996511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:17.473217 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:17.473189 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:20.047396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:20.047355 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 19:30:20.047789 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:20.047442 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 19:30:21.105363 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:21.105324 2582 generic.go:356] "Generic (PLEG): container finished" podID="90d484ae-d2af-482b-8622-9a61f4d96461" containerID="39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9" exitCode=0 Jul 15 19:30:21.105988 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:21.105961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" event={"ID":"90d484ae-d2af-482b-8622-9a61f4d96461","Type":"ContainerDied","Data":"39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9"} Jul 15 19:30:21.106572 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:21.106553 2582 scope.go:117] "RemoveContainer" containerID="39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9" Jul 15 19:30:21.108787 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:21.108743 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:21.108933 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:21.108848 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:21.108933 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:21.108896 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:22.119223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:22.119186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" event={"ID":"90d484ae-d2af-482b-8622-9a61f4d96461","Type":"ContainerStarted","Data":"3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811"} Jul 15 19:30:22.853157 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:22.853122 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:22.853369 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:22.853229 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 19:30:22.853369 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:22.853303 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 19:30:23.853497 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:23.853460 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 19:30:23.853869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:23.853512 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 19:30:24.853297 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:24.853259 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 19:30:24.853473 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:24.853316 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 19:30:25.696003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:25.695932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:25.795960 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:25.795924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:25.853100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:25.853067 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 19:30:25.853323 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:25.853119 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 19:30:26.857260 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:26.857214 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:26.857717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:26.857704 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:26.858336 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:26.858317 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:30:27.608488 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:27.607688 2582 patch_prober.go:28] interesting pod/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:44166->10.133.0.61:15021: read: connection reset by peer" start-of-body= Jul 15 19:30:27.608488 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:27.607742 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:44166->10.133.0.61:15021: read: connection reset by peer" Jul 15 19:30:28.157978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:28.157949 2582 generic.go:356] "Generic (PLEG): container finished" podID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerID="9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2" exitCode=0 Jul 15 19:30:28.158436 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:28.158025 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" event={"ID":"f532e1c9-22b8-406d-97c4-d621762a8cbe","Type":"ContainerDied","Data":"9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2"} Jul 15 19:30:28.158514 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:28.158500 2582 scope.go:117] "RemoveContainer" containerID="9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2" Jul 15 19:30:28.159444 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:28.159416 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:28.159559 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:28.159493 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:28.159559 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:28.159539 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:30:29.165396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:29.165356 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" event={"ID":"f532e1c9-22b8-406d-97c4-d621762a8cbe","Type":"ContainerStarted","Data":"c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f"} Jul 15 19:30:29.318988 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:29.318950 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:29.319150 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:29.319112 2582 patch_prober.go:28] interesting pod/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 15 19:30:29.319194 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:29.319152 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 15 19:30:30.318682 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:30.318644 2582 patch_prober.go:28] interesting pod/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 15 19:30:30.319073 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:30.318718 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 15 19:30:31.163757 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:31.163721 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:31.319028 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:31.318984 2582 patch_prober.go:28] interesting pod/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 15 19:30:31.319538 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:31.319058 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 15 19:30:32.318756 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:32.318722 2582 patch_prober.go:28] interesting pod/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 15 19:30:32.318937 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:32.318791 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 15 19:30:33.322176 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:33.322143 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:33.322643 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:33.322510 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:33.323227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:33.323209 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:30:34.898179 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:34.898144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:34.998685 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:34.998644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:48.216785 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:48.216759 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:30:48.221884 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:48.221865 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:30:51.450293 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:51.450261 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:51.642782 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:30:51.642746 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:02.441768 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:31:02.441732 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:26.133745 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:31:26.133707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:28.755330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:31:28.755275 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:32.535349 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:31:32.535316 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:59.298174 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:31:59.298139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:59.398204 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:31:59.398172 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:04.926227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:04.926154 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:05.026932 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:05.026899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:05.181140 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:05.181055 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp"] Jul 15 19:32:05.181420 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:05.181324 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" containerID="cri-o://3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811" gracePeriod=30 Jul 15 19:32:05.219204 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:05.219175 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:05.416777 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:05.416747 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:05.863856 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:05.863819 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:09.528978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:09.528945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:09.584378 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:09.584348 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:10.448799 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.448742 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:32:10.524177 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524149 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d484ae-d2af-482b-8622-9a61f4d96461-istiod-ca-cert\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524177 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524180 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-envoy\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524214 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-credential-socket\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524263 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-istio-token\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524305 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-socket\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524320 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-data\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524385 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-certs\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524411 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdq72\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-kube-api-access-fdq72\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524459 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d484ae-d2af-482b-8622-9a61f4d96461-istio-podinfo\") pod \"90d484ae-d2af-482b-8622-9a61f4d96461\" (UID: \"90d484ae-d2af-482b-8622-9a61f4d96461\") " Jul 15 19:32:10.524833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:10.524833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524622 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d484ae-d2af-482b-8622-9a61f4d96461-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:32:10.524833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524616 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:10.525050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524834 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d484ae-d2af-482b-8622-9a61f4d96461-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.525050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524855 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.525050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524871 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.525050 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.524867 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:10.525339 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.525303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-data" (OuterVolumeSpecName: "istio-data") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:10.526538 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.526512 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:10.526636 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.526591 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90d484ae-d2af-482b-8622-9a61f4d96461-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:32:10.526717 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.526696 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-kube-api-access-fdq72" (OuterVolumeSpecName: "kube-api-access-fdq72") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "kube-api-access-fdq72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:10.526769 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.526729 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-istio-token" (OuterVolumeSpecName: "istio-token") pod "90d484ae-d2af-482b-8622-9a61f4d96461" (UID: "90d484ae-d2af-482b-8622-9a61f4d96461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:10.625579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.625539 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d484ae-d2af-482b-8622-9a61f4d96461-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.625579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.625569 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.625579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.625578 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.625579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.625586 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.626031 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.625595 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d484ae-d2af-482b-8622-9a61f4d96461-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.626031 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.625603 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdq72\" (UniqueName: \"kubernetes.io/projected/90d484ae-d2af-482b-8622-9a61f4d96461-kube-api-access-fdq72\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:10.696801 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.696765 2582 generic.go:356] "Generic (PLEG): container finished" podID="90d484ae-d2af-482b-8622-9a61f4d96461" containerID="3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811" exitCode=0 Jul 15 19:32:10.697003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.696816 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" Jul 15 19:32:10.697003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.696854 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" event={"ID":"90d484ae-d2af-482b-8622-9a61f4d96461","Type":"ContainerDied","Data":"3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811"} Jul 15 19:32:10.697003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.696916 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp" event={"ID":"90d484ae-d2af-482b-8622-9a61f4d96461","Type":"ContainerDied","Data":"1fa6be596cf00efaa8ae7e7955003f48c0ad2fc75851a913fa3d79c9e380a8b1"} Jul 15 19:32:10.697003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.696941 2582 scope.go:117] "RemoveContainer" containerID="3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811" Jul 15 19:32:10.707579 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.707562 2582 scope.go:117] "RemoveContainer" containerID="39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9" Jul 15 19:32:10.715819 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.715796 2582 scope.go:117] "RemoveContainer" containerID="3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811" Jul 15 19:32:10.716072 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:32:10.716051 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811\": container with ID starting with 3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811 not found: ID does not exist" containerID="3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811" Jul 15 19:32:10.716122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.716081 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811"} err="failed to get container status \"3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811\": rpc error: code = NotFound desc = could not find container \"3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811\": container with ID starting with 3bf60dcfa38336dd6e4a6947394de3b926604bd2d9f0b8799546c283a53fa811 not found: ID does not exist" Jul 15 19:32:10.716122 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.716099 2582 scope.go:117] "RemoveContainer" containerID="39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9" Jul 15 19:32:10.716387 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:32:10.716365 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9\": container with ID starting with 39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9 not found: ID does not exist" containerID="39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9" Jul 15 19:32:10.716455 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.716397 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9"} err="failed to get container status \"39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9\": rpc error: code = NotFound desc = could not find container \"39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9\": container with ID starting with 39afe907e1aaa705dfa222864ee57a529236b913477eb57bf05ac383db429eb9 not found: ID does not exist" Jul 15 19:32:10.720004 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.719983 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp"] Jul 15 19:32:10.724080 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:10.724061 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4256e5-openshift-default-745f59868c-xwbpp"] Jul 15 19:32:11.640908 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:11.640870 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:12.199730 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:12.199697 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" path="/var/lib/kubelet/pods/90d484ae-d2af-482b-8622-9a61f4d96461/volumes" Jul 15 19:32:12.427586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:12.427548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:18.218384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:18.218318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:18.318286 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:18.318255 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:18.471338 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:18.471308 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf"] Jul 15 19:32:18.471543 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:18.471521 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" containerID="cri-o://c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f" gracePeriod=30 Jul 15 19:32:18.495198 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:18.495171 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:18.743614 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:18.743528 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:19.117652 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.117559 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:19.398745 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.398670 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn"] Jul 15 19:32:19.399083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.399036 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" Jul 15 19:32:19.399083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.399046 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" Jul 15 19:32:19.399083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.399066 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" Jul 15 19:32:19.399083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.399072 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" Jul 15 19:32:19.399216 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.399124 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" Jul 15 19:32:19.402268 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.402235 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.406319 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.406298 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4293f0-openshift-default-dockercfg-8zd28\"" Jul 15 19:32:19.429211 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.429186 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn"] Jul 15 19:32:19.441872 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.441846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:19.504952 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.504898 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-token\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.504952 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.504952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istiod-ca-cert\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.505227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.504983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-certs\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.505227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.505042 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-podinfo\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.505227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.505079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-socket\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.505227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.505148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-envoy\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.505227 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.505199 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-credential-socket\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.505532 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.505296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8bn7\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-kube-api-access-f8bn7\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.505532 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.505374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-data\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.606621 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-token\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.606621 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606619 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istiod-ca-cert\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.606866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-certs\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.606866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-podinfo\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.606866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-socket\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.606866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-envoy\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.606866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-credential-socket\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.607132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8bn7\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-kube-api-access-f8bn7\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.607132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.606925 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-data\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.607261 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.607130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-certs\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.607389 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.607369 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-data\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.607453 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.607392 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-socket\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.607529 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.607511 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-credential-socket\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.607569 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.607526 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istiod-ca-cert\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.609585 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.609555 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-envoy\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.609695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.609679 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-podinfo\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.623598 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.623575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-token\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.624127 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.624109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8bn7\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-kube-api-access-f8bn7\") pod \"e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.712516 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.712433 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:19.753043 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.753004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:19.874627 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.874603 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn"] Jul 15 19:32:19.877937 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:32:19.877910 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485b0ca1_5ab1_4854_81d3_5ad4dd6b7c5e.slice/crio-96a686ddf74646059c7a76dce73245480eb23921cf8cefbfc162d5746c9753d9 WatchSource:0}: Error finding container 96a686ddf74646059c7a76dce73245480eb23921cf8cefbfc162d5746c9753d9: Status 404 returned error can't find the container with id 96a686ddf74646059c7a76dce73245480eb23921cf8cefbfc162d5746c9753d9 Jul 15 19:32:19.880310 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.880268 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:19.880429 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.880360 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:19.880429 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:19.880400 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:20.755463 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:20.755424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" event={"ID":"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e","Type":"ContainerStarted","Data":"fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d"} Jul 15 19:32:20.755861 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:20.755483 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" event={"ID":"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e","Type":"ContainerStarted","Data":"96a686ddf74646059c7a76dce73245480eb23921cf8cefbfc162d5746c9753d9"} Jul 15 19:32:20.777665 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:20.777615 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" podStartSLOduration=1.777599828 podStartE2EDuration="1.777599828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:20.775816135 +0000 UTC m=+1893.171172106" watchObservedRunningTime="2026-07-15 19:32:20.777599828 +0000 UTC m=+1893.172955797" Jul 15 19:32:21.694716 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:21.694684 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:21.712841 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:21.712790 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:21.718693 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:21.718669 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:21.761095 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:21.761069 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:21.761906 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:21.761890 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:21.772528 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:21.772502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:21.973384 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:21.973356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:22.278170 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:22.278079 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:22.586589 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:22.586496 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:22.692960 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:22.692928 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:23.723542 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.723517 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:32:23.772633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.772542 2582 generic.go:356] "Generic (PLEG): container finished" podID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerID="c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f" exitCode=0 Jul 15 19:32:23.772633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.772614 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" Jul 15 19:32:23.772838 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.772637 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" event={"ID":"f532e1c9-22b8-406d-97c4-d621762a8cbe","Type":"ContainerDied","Data":"c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f"} Jul 15 19:32:23.772838 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.772690 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf" event={"ID":"f532e1c9-22b8-406d-97c4-d621762a8cbe","Type":"ContainerDied","Data":"e71137b345565dbca51693bc39a797a04749261fd8fba06ce7d80ca80afc66f3"} Jul 15 19:32:23.772838 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.772711 2582 scope.go:117] "RemoveContainer" containerID="c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f" Jul 15 19:32:23.781673 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.781655 2582 scope.go:117] "RemoveContainer" containerID="9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2" Jul 15 19:32:23.790299 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.790282 2582 scope.go:117] "RemoveContainer" containerID="c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f" Jul 15 19:32:23.790569 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:32:23.790554 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f\": container with ID starting with c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f not found: ID does not exist" containerID="c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f" Jul 15 19:32:23.790616 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.790577 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f"} err="failed to get container status \"c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f\": rpc error: code = NotFound desc = could not find container \"c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f\": container with ID starting with c0258d8a4857273517cc24a851276af7d9caddfb38333580534f7f571d300e9f not found: ID does not exist" Jul 15 19:32:23.790616 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.790592 2582 scope.go:117] "RemoveContainer" containerID="9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2" Jul 15 19:32:23.790796 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:32:23.790779 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2\": container with ID starting with 9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2 not found: ID does not exist" containerID="9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2" Jul 15 19:32:23.790839 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.790800 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2"} err="failed to get container status \"9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2\": rpc error: code = NotFound desc = could not find container \"9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2\": container with ID starting with 9da5b1b9ec28c401f040f378d62cba899fd3a5abd85b222301f7d86805583ff2 not found: ID does not exist" Jul 15 19:32:23.849194 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849152 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-socket\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849207 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-podinfo\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849268 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-credential-socket\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849302 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-data\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849331 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-envoy\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849357 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-certs\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849372 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849373 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f532e1c9-22b8-406d-97c4-d621762a8cbe-istiod-ca-cert\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849674 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849403 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-token\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849674 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849461 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xhwh\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-kube-api-access-2xhwh\") pod \"f532e1c9-22b8-406d-97c4-d621762a8cbe\" (UID: \"f532e1c9-22b8-406d-97c4-d621762a8cbe\") " Jul 15 19:32:23.849674 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849514 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:23.849674 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849590 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:23.849900 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849809 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.849900 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849832 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.850014 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.849937 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f532e1c9-22b8-406d-97c4-d621762a8cbe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:32:23.850014 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.850002 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:23.850474 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.850444 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-data" (OuterVolumeSpecName: "istio-data") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:23.851587 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.851556 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:32:23.851682 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.851639 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-kube-api-access-2xhwh" (OuterVolumeSpecName: "kube-api-access-2xhwh") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "kube-api-access-2xhwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:23.851999 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.851979 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:23.852055 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.851990 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-token" (OuterVolumeSpecName: "istio-token") pod "f532e1c9-22b8-406d-97c4-d621762a8cbe" (UID: "f532e1c9-22b8-406d-97c4-d621762a8cbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:23.950569 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.950529 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.950569 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.950563 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.950569 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.950577 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f532e1c9-22b8-406d-97c4-d621762a8cbe-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.950869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.950591 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f532e1c9-22b8-406d-97c4-d621762a8cbe-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.950869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.950606 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.950869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.950622 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xhwh\" (UniqueName: \"kubernetes.io/projected/f532e1c9-22b8-406d-97c4-d621762a8cbe-kube-api-access-2xhwh\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:23.950869 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:23.950637 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f532e1c9-22b8-406d-97c4-d621762a8cbe-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:32:24.096431 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:24.096400 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf"] Jul 15 19:32:24.100499 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:24.100474 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-381b50-openshift-default-6cbb69bc7-w2krf"] Jul 15 19:32:24.206026 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:24.205992 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" path="/var/lib/kubelet/pods/f532e1c9-22b8-406d-97c4-d621762a8cbe/volumes" Jul 15 19:32:27.190424 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.190387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:27.632650 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.632610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:27.912194 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.912101 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7"] Jul 15 19:32:27.912565 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.912549 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" Jul 15 19:32:27.912565 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.912566 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" Jul 15 19:32:27.912695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.912637 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" Jul 15 19:32:27.912695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.912647 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d484ae-d2af-482b-8622-9a61f4d96461" containerName="istio-proxy" Jul 15 19:32:27.915608 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.915593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.917975 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.917955 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a115a3-openshift-default-dockercfg-ds5fz\"" Jul 15 19:32:27.928502 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.928476 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7"] Jul 15 19:32:27.950013 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.949933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:27.988500 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-socket\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988500 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-envoy\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988525 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-podinfo\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-token\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5mr\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-kube-api-access-ql5mr\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-certs\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-credential-socket\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istiod-ca-cert\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:27.988868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:27.988802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-data\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.090104 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.090051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-credential-socket\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.090421 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.090393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istiod-ca-cert\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.090610 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.090591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-data\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.090767 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.090750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-socket\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.090872 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.090858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-envoy\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.090995 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.090982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-podinfo\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.091120 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.091107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-token\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.091231 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.091217 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5mr\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-kube-api-access-ql5mr\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.091358 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.091340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-certs\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.091992 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.091964 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-certs\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.092400 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.092378 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-credential-socket\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.093147 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.093125 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istiod-ca-cert\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.093934 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.093914 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-data\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.094311 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.094292 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-socket\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.099607 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.099576 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-envoy\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.101802 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.099776 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-podinfo\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.104587 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.104552 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-token\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.108259 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.106485 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5mr\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-kube-api-access-ql5mr\") pod \"e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.230991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.230959 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:28.320371 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.319918 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:28.410596 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.410565 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7"] Jul 15 19:32:28.413209 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:32:28.413184 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617734ef_aeb2_42a3_8284_e4cfdecf6b8b.slice/crio-82a1af722f946012076a9605337c51edd2b6e68e5442c343dd17c6fc66e5244e WatchSource:0}: Error finding container 82a1af722f946012076a9605337c51edd2b6e68e5442c343dd17c6fc66e5244e: Status 404 returned error can't find the container with id 82a1af722f946012076a9605337c51edd2b6e68e5442c343dd17c6fc66e5244e Jul 15 19:32:28.415416 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.415385 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:28.415496 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.415451 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:28.415496 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.415484 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:28.809163 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.809119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" event={"ID":"617734ef-aeb2-42a3-8284-e4cfdecf6b8b","Type":"ContainerStarted","Data":"72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372"} Jul 15 19:32:28.809371 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.809189 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" event={"ID":"617734ef-aeb2-42a3-8284-e4cfdecf6b8b","Type":"ContainerStarted","Data":"82a1af722f946012076a9605337c51edd2b6e68e5442c343dd17c6fc66e5244e"} Jul 15 19:32:28.831774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:28.831714 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podStartSLOduration=1.831695138 podStartE2EDuration="1.831695138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:28.829569647 +0000 UTC m=+1901.224925615" watchObservedRunningTime="2026-07-15 19:32:28.831695138 +0000 UTC m=+1901.227051110" Jul 15 19:32:29.231576 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:29.231534 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:29.236809 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:29.236785 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:29.713330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:29.713253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:29.809037 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:29.808996 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:29.815132 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:29.815104 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:29.816022 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:29.816006 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:29.916116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:29.916086 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:30.205874 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:30.205819 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:30.434363 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:30.434315 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:30.534171 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:30.534131 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:35.211554 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:35.211523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:35.848203 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:35.848168 2582 generic.go:356] "Generic (PLEG): container finished" podID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerID="fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d" exitCode=0 Jul 15 19:32:35.848479 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:35.848263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" event={"ID":"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e","Type":"ContainerDied","Data":"fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d"} Jul 15 19:32:35.848687 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:35.848672 2582 scope.go:117] "RemoveContainer" containerID="fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d" Jul 15 19:32:35.849760 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:35.849731 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:35.849825 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:35.849811 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:35.849861 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:35.849850 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:36.761268 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:36.761219 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:36.855697 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:36.855664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" event={"ID":"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e","Type":"ContainerStarted","Data":"274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4"} Jul 15 19:32:37.712822 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:37.712778 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:37.714075 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:37.714045 2582 patch_prober.go:28] interesting pod/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 15 19:32:37.714230 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:37.714093 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 15 19:32:38.713416 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:38.713380 2582 patch_prober.go:28] interesting pod/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 15 19:32:38.713826 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:38.713436 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 15 19:32:39.713046 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:39.712998 2582 patch_prober.go:28] interesting pod/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 15 19:32:39.713269 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:39.713068 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 15 19:32:40.717154 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:40.717116 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:40.717611 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:40.717537 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:40.718452 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:40.718430 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:32:43.498083 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:43.498036 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:43.597848 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:43.597801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:43.697846 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:43.697812 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:45.856435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:45.856390 2582 patch_prober.go:28] interesting pod/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 19:32:45.856833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:45.856468 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 19:32:45.919404 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:45.919376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:46.922637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:46.922547 2582 generic.go:356] "Generic (PLEG): container finished" podID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerID="72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372" exitCode=0 Jul 15 19:32:46.922637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:46.922619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" event={"ID":"617734ef-aeb2-42a3-8284-e4cfdecf6b8b","Type":"ContainerDied","Data":"72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372"} Jul 15 19:32:46.923086 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:46.923069 2582 scope.go:117] "RemoveContainer" containerID="72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372" Jul 15 19:32:46.924129 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:46.924099 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:46.924266 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:46.924162 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:46.924266 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:46.924192 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:32:47.930141 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:47.930106 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" event={"ID":"617734ef-aeb2-42a3-8284-e4cfdecf6b8b","Type":"ContainerStarted","Data":"86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e"} Jul 15 19:32:48.231686 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:48.231630 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:48.232202 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:48.232170 2582 patch_prober.go:28] interesting pod/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 19:32:48.232329 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:48.232231 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 19:32:48.975232 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:48.975198 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:49.231426 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:49.231342 2582 patch_prober.go:28] interesting pod/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 19:32:49.231426 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:49.231396 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 19:32:49.355945 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:49.355916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:50.231378 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:50.231343 2582 patch_prober.go:28] interesting pod/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 19:32:50.231771 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:50.231395 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 19:32:51.232376 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:51.232331 2582 patch_prober.go:28] interesting pod/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 19:32:51.232883 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:51.232393 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 19:32:51.314464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:51.314432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:51.414439 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:51.414405 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:52.231396 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:52.231361 2582 patch_prober.go:28] interesting pod/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 19:32:52.231574 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:52.231418 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 19:32:53.235089 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:53.235059 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:53.235497 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:53.235442 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:53.236179 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:53.236164 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:32:54.901441 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:54.901397 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:55.002573 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.002542 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:55.175648 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.175575 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn"] Jul 15 19:32:55.175808 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.175784 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" containerID="cri-o://274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4" gracePeriod=30 Jul 15 19:32:55.198976 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.198943 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:55.406920 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.406888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:55.719794 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.719757 2582 patch_prober.go:28] interesting pod/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:32:55.719964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.719815 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:32:55.841019 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:32:55.840983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:00.438788 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.438729 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:33:00.515337 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515307 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-token\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515539 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515375 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-data\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515539 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515400 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-envoy\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515539 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515427 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8bn7\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-kube-api-access-f8bn7\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515539 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515459 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-socket\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515539 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515533 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-podinfo\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515615 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-credential-socket\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515675 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-certs\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515712 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istiod-ca-cert\") pod \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\" (UID: \"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e\") " Jul 15 19:33:00.515811 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.515759 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:00.516088 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.516042 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.516088 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.516045 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:00.516088 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.516061 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:00.516362 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.516334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-data" (OuterVolumeSpecName: "istio-data") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:00.516411 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.516389 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:33:00.517704 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.517673 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:33:00.517808 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.517741 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:00.518133 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.518111 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-kube-api-access-f8bn7" (OuterVolumeSpecName: "kube-api-access-f8bn7") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "kube-api-access-f8bn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:33:00.518190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.518114 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" (UID: "485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:33:00.616878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616844 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.616878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616877 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.616878 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616886 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.617116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616895 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.617116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616903 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.617116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616911 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.617116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616919 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8bn7\" (UniqueName: \"kubernetes.io/projected/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-kube-api-access-f8bn7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.617116 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.616928 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:00.916760 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:00.916727 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:01.006469 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.006431 2582 generic.go:356] "Generic (PLEG): container finished" podID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerID="274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4" exitCode=0 Jul 15 19:33:01.006636 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.006481 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" Jul 15 19:33:01.006636 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.006520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" event={"ID":"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e","Type":"ContainerDied","Data":"274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4"} Jul 15 19:33:01.006636 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.006579 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn" event={"ID":"485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e","Type":"ContainerDied","Data":"96a686ddf74646059c7a76dce73245480eb23921cf8cefbfc162d5746c9753d9"} Jul 15 19:33:01.006636 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.006609 2582 scope.go:117] "RemoveContainer" containerID="274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4" Jul 15 19:33:01.016219 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.016202 2582 scope.go:117] "RemoveContainer" containerID="fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d" Jul 15 19:33:01.024258 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.024230 2582 scope.go:117] "RemoveContainer" containerID="274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4" Jul 15 19:33:01.024504 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:33:01.024486 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4\": container with ID starting with 274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4 not found: ID does not exist" containerID="274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4" Jul 15 19:33:01.024570 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.024514 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4"} err="failed to get container status \"274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4\": rpc error: code = NotFound desc = could not find container \"274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4\": container with ID starting with 274ec83b175a58eaeeec7c4a54e3994fc53f34dbf4bad2fd1ab7fd6eb2c201b4 not found: ID does not exist" Jul 15 19:33:01.024570 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.024530 2582 scope.go:117] "RemoveContainer" containerID="fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d" Jul 15 19:33:01.024767 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:33:01.024751 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d\": container with ID starting with fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d not found: ID does not exist" containerID="fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d" Jul 15 19:33:01.024806 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.024776 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d"} err="failed to get container status \"fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d\": rpc error: code = NotFound desc = could not find container \"fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d\": container with ID starting with fd558c2287614b3a28db668ae3b993f7373eaea01749490021d613760c962b4d not found: ID does not exist" Jul 15 19:33:01.028984 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.028962 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn"] Jul 15 19:33:01.033748 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.033726 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4293f0-openshift-default-679d9c84b7-4qsjn"] Jul 15 19:33:01.969475 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:01.969444 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:02.202451 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:02.200646 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" path="/var/lib/kubelet/pods/485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e/volumes" Jul 15 19:33:07.469306 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:07.469221 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:07.571433 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:07.571399 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:07.725316 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:07.725201 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7"] Jul 15 19:33:07.725507 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:07.725458 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" containerID="cri-o://86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e" gracePeriod=30 Jul 15 19:33:07.744426 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:07.744396 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:07.951824 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:07.951788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:08.237941 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.237903 2582 patch_prober.go:28] interesting pod/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:33:08.238186 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.237977 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:33:08.290323 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.290287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:08.558034 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.557946 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl"] Jul 15 19:33:08.558560 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558540 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" Jul 15 19:33:08.558610 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558562 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" Jul 15 19:33:08.558610 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558573 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" Jul 15 19:33:08.558610 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558581 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" Jul 15 19:33:08.558610 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558592 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" Jul 15 19:33:08.558610 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558599 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" Jul 15 19:33:08.558764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558702 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f532e1c9-22b8-406d-97c4-d621762a8cbe" containerName="istio-proxy" Jul 15 19:33:08.558764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558714 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" Jul 15 19:33:08.558764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.558725 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="485b0ca1-5ab1-4854-81d3-5ad4dd6b7c5e" containerName="istio-proxy" Jul 15 19:33:08.562098 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.562077 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.565422 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.565398 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-4a4ed2-openshift-default-dockercfg-tc4rk\"" Jul 15 19:33:08.576292 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.576264 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl"] Jul 15 19:33:08.591494 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.591459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:08.691090 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-podinfo\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691113 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-credential-socket\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-envoy\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691298 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-data\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2z9b\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-kube-api-access-n2z9b\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-token\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istiod-ca-cert\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691434 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-certs\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.691594 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.691477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-socket\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.791973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.791942 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-token\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.791973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.791978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istiod-ca-cert\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.791998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-certs\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-socket\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-podinfo\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-credential-socket\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-envoy\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-data\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792225 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2z9b\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-kube-api-access-n2z9b\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792632 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792573 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-credential-socket\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-data\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792637 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-certs\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-socket\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.792866 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.792847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istiod-ca-cert\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.794652 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.794632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-envoy\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.794779 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.794760 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-podinfo\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.805348 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.805318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-token\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.806145 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.806125 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2z9b\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-kube-api-access-n2z9b\") pod \"e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:08.809199 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.809149 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:08.872320 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:08.872287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:09.008806 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:09.008776 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl"] Jul 15 19:33:09.010307 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:33:09.010284 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f8ead7_e65c_4403_ac4a_63b69cb39ef4.slice/crio-50cd0d95c17f297fcbdba288086e9676a3c099c2a5112633d20d3f26e992ce76 WatchSource:0}: Error finding container 50cd0d95c17f297fcbdba288086e9676a3c099c2a5112633d20d3f26e992ce76: Status 404 returned error can't find the container with id 50cd0d95c17f297fcbdba288086e9676a3c099c2a5112633d20d3f26e992ce76 Jul 15 19:33:09.012303 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:09.012265 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:09.012361 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:09.012341 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:09.012406 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:09.012372 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:09.052513 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:09.052486 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" event={"ID":"15f8ead7-e65c-4403-ac4a-63b69cb39ef4","Type":"ContainerStarted","Data":"50cd0d95c17f297fcbdba288086e9676a3c099c2a5112633d20d3f26e992ce76"} Jul 15 19:33:10.059265 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:10.059194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" event={"ID":"15f8ead7-e65c-4403-ac4a-63b69cb39ef4","Type":"ContainerStarted","Data":"dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487"} Jul 15 19:33:10.082520 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:10.082467 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" podStartSLOduration=2.082452148 podStartE2EDuration="2.082452148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:33:10.08003576 +0000 UTC m=+1942.475391730" watchObservedRunningTime="2026-07-15 19:33:10.082452148 +0000 UTC m=+1942.477808118" Jul 15 19:33:10.872983 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:10.872945 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:10.877934 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:10.877913 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:10.984491 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:10.984461 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:11.066388 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:11.066346 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:11.073393 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:11.073365 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:11.083449 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:11.083422 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:11.198638 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:11.198552 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:11.633646 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:11.633605 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:11.802829 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:11.802798 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:11.901823 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:11.901740 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:12.996275 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:12.996207 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:33:13.077462 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.077429 2582 generic.go:356] "Generic (PLEG): container finished" podID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerID="86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e" exitCode=0 Jul 15 19:33:13.077631 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.077482 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" Jul 15 19:33:13.077631 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.077486 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" event={"ID":"617734ef-aeb2-42a3-8284-e4cfdecf6b8b","Type":"ContainerDied","Data":"86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e"} Jul 15 19:33:13.077631 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.077540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7" event={"ID":"617734ef-aeb2-42a3-8284-e4cfdecf6b8b","Type":"ContainerDied","Data":"82a1af722f946012076a9605337c51edd2b6e68e5442c343dd17c6fc66e5244e"} Jul 15 19:33:13.077631 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.077563 2582 scope.go:117] "RemoveContainer" containerID="86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e" Jul 15 19:33:13.086951 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.086930 2582 scope.go:117] "RemoveContainer" containerID="72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372" Jul 15 19:33:13.095377 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.095353 2582 scope.go:117] "RemoveContainer" containerID="86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e" Jul 15 19:33:13.095629 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:33:13.095610 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e\": container with ID starting with 86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e not found: ID does not exist" containerID="86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e" Jul 15 19:33:13.095700 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.095639 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e"} err="failed to get container status \"86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e\": rpc error: code = NotFound desc = could not find container \"86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e\": container with ID starting with 86539b48916edd136cee83925397f8c284a72600e719ecaeb49cb8747bc6917e not found: ID does not exist" Jul 15 19:33:13.095700 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.095656 2582 scope.go:117] "RemoveContainer" containerID="72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372" Jul 15 19:33:13.095913 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:33:13.095894 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372\": container with ID starting with 72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372 not found: ID does not exist" containerID="72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372" Jul 15 19:33:13.095957 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.095920 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372"} err="failed to get container status \"72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372\": rpc error: code = NotFound desc = could not find container \"72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372\": container with ID starting with 72f36dabe4af7534a668a8c2ab6a5598eee8e513cf24870968b92ae721a95372 not found: ID does not exist" Jul 15 19:33:13.135475 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135447 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql5mr\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-kube-api-access-ql5mr\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135505 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-token\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135637 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135529 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istiod-ca-cert\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135661 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-data\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135737 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-certs\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135766 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-podinfo\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135798 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-credential-socket\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135848 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-envoy\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135893 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-socket\") pod \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\" (UID: \"617734ef-aeb2-42a3-8284-e4cfdecf6b8b\") " Jul 15 19:33:13.135978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.135916 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:33:13.136353 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.136008 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:13.136353 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.136233 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.136353 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.136271 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.136511 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.136455 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:13.136559 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.136526 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:13.136661 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.136630 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-data" (OuterVolumeSpecName: "istio-data") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:13.137879 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.137857 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:33:13.137999 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.137884 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-kube-api-access-ql5mr" (OuterVolumeSpecName: "kube-api-access-ql5mr") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "kube-api-access-ql5mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:33:13.138070 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.138012 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:33:13.138264 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.138229 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "617734ef-aeb2-42a3-8284-e4cfdecf6b8b" (UID: "617734ef-aeb2-42a3-8284-e4cfdecf6b8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:13.237586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.237547 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5mr\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-kube-api-access-ql5mr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.237586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.237578 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.237586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.237591 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.237826 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.237603 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.237826 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.237614 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.237826 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.237625 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.237826 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.237636 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617734ef-aeb2-42a3-8284-e4cfdecf6b8b-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:33:13.401486 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.401456 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7"] Jul 15 19:33:13.406079 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:13.406056 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a115a3-openshift-default-99cb94b7c-kb9m7"] Jul 15 19:33:14.204521 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:14.204482 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" path="/var/lib/kubelet/pods/617734ef-aeb2-42a3-8284-e4cfdecf6b8b/volumes" Jul 15 19:33:16.478219 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:16.478190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:16.818110 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:16.818003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:17.174405 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.174295 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6"] Jul 15 19:33:17.174807 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.174788 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" Jul 15 19:33:17.174894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.174809 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" Jul 15 19:33:17.174894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.174825 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" Jul 15 19:33:17.174894 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.174832 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" Jul 15 19:33:17.175052 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.174917 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" Jul 15 19:33:17.175052 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.174930 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="617734ef-aeb2-42a3-8284-e4cfdecf6b8b" containerName="istio-proxy" Jul 15 19:33:17.178024 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.178002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.183713 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.183689 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f83827-openshift-default-dockercfg-2mpnv\"" Jul 15 19:33:17.238395 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.238362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:17.271474 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-data\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77717dc8-3b93-4d11-bb74-d774af63f642-istiod-ca-cert\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-credential-socket\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77717dc8-3b93-4d11-bb74-d774af63f642-istio-podinfo\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271558 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5htkr\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-kube-api-access-5htkr\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271639 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-istio-token\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-socket\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-certs\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.271865 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.271713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-envoy\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.302496 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.302436 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6"] Jul 15 19:33:17.372848 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.372801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-socket\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.372848 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.372853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-certs\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.372873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-envoy\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.372913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-data\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.372941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77717dc8-3b93-4d11-bb74-d774af63f642-istiod-ca-cert\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.372980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-credential-socket\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77717dc8-3b93-4d11-bb74-d774af63f642-istio-podinfo\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5htkr\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-kube-api-access-5htkr\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373118 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-istio-token\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373559 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373308 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-socket\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373559 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373377 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-data\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373559 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373449 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-credential-socket\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373688 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-certs\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.373727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.373718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77717dc8-3b93-4d11-bb74-d774af63f642-istiod-ca-cert\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.375474 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.375448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-envoy\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.375675 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.375654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77717dc8-3b93-4d11-bb74-d774af63f642-istio-podinfo\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.387049 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.384682 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5htkr\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-kube-api-access-5htkr\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.395782 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.395720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-istio-token\") pod \"e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.425160 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.425072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:17.502344 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.502299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:17.663415 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.663382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6"] Jul 15 19:33:17.664595 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:33:17.664570 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77717dc8_3b93_4d11_bb74_d774af63f642.slice/crio-854fbd0ef2e595c7684f8fc63da2b9c4e62c2a66a85708e8090e39c3abb6d7b2 WatchSource:0}: Error finding container 854fbd0ef2e595c7684f8fc63da2b9c4e62c2a66a85708e8090e39c3abb6d7b2: Status 404 returned error can't find the container with id 854fbd0ef2e595c7684f8fc63da2b9c4e62c2a66a85708e8090e39c3abb6d7b2 Jul 15 19:33:17.666745 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.666717 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:17.666813 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.666783 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:17.666848 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:17.666812 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:18.117039 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:18.117006 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" event={"ID":"77717dc8-3b93-4d11-bb74-d774af63f642","Type":"ContainerStarted","Data":"2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14"} Jul 15 19:33:18.117214 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:18.117066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" event={"ID":"77717dc8-3b93-4d11-bb74-d774af63f642","Type":"ContainerStarted","Data":"854fbd0ef2e595c7684f8fc63da2b9c4e62c2a66a85708e8090e39c3abb6d7b2"} Jul 15 19:33:18.141478 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:18.141427 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podStartSLOduration=1.141411416 podStartE2EDuration="1.141411416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:33:18.138903612 +0000 UTC m=+1950.534259598" watchObservedRunningTime="2026-07-15 19:33:18.141411416 +0000 UTC m=+1950.536767385" Jul 15 19:33:18.503797 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:18.503763 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:18.508978 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:18.508953 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:19.020232 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.020201 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:19.116325 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.116294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:19.123385 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.123357 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:19.124347 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.124325 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:19.193843 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.193793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:19.383868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.383776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:19.754217 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.754187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:19.853764 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:19.853663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:21.135412 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:21.135328 2582 generic.go:356] "Generic (PLEG): container finished" podID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerID="dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487" exitCode=0 Jul 15 19:33:21.135891 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:21.135405 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" event={"ID":"15f8ead7-e65c-4403-ac4a-63b69cb39ef4","Type":"ContainerDied","Data":"dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487"} Jul 15 19:33:21.135982 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:21.135963 2582 scope.go:117] "RemoveContainer" containerID="dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487" Jul 15 19:33:21.136973 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:21.136946 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:21.137041 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:21.137011 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:21.137081 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:21.137047 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:22.142306 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:22.142272 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" event={"ID":"15f8ead7-e65c-4403-ac4a-63b69cb39ef4","Type":"ContainerStarted","Data":"b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d"} Jul 15 19:33:22.872516 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:22.872480 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:22.873328 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:22.873304 2582 patch_prober.go:28] interesting pod/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 15 19:33:22.873399 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:22.873368 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 15 19:33:23.872664 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:23.872628 2582 patch_prober.go:28] interesting pod/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 15 19:33:23.873111 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:23.872689 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 15 19:33:24.500892 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:24.500806 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:24.873664 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:24.873580 2582 patch_prober.go:28] interesting pod/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 15 19:33:24.873664 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:24.873631 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 15 19:33:25.873377 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:25.873343 2582 patch_prober.go:28] interesting pod/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 15 19:33:25.873556 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:25.873396 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 15 19:33:26.066966 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:26.066932 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:26.876581 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:26.876553 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:26.877465 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:26.877445 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:33:32.615906 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:32.615872 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:32.715581 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:32.715547 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:35.165427 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:35.165379 2582 patch_prober.go:28] interesting pod/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 19:33:35.165887 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:35.165456 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 19:33:36.224550 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:36.224514 2582 generic.go:356] "Generic (PLEG): container finished" podID="77717dc8-3b93-4d11-bb74-d774af63f642" containerID="2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14" exitCode=0 Jul 15 19:33:36.224994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:36.224584 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" event={"ID":"77717dc8-3b93-4d11-bb74-d774af63f642","Type":"ContainerDied","Data":"2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14"} Jul 15 19:33:36.224994 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:36.224974 2582 scope.go:117] "RemoveContainer" containerID="2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14" Jul 15 19:33:36.225974 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:36.225948 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:36.226038 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:36.226016 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:36.226074 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:36.226057 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 15 19:33:37.231775 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:37.231742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" event={"ID":"77717dc8-3b93-4d11-bb74-d774af63f642","Type":"ContainerStarted","Data":"e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896"} Jul 15 19:33:37.503409 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:37.503300 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:37.503409 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:37.503385 2582 patch_prober.go:28] interesting pod/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 19:33:37.503662 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:37.503439 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 19:33:38.503113 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:38.503055 2582 patch_prober.go:28] interesting pod/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 19:33:38.503757 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:38.503133 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 19:33:39.503283 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:39.503218 2582 patch_prober.go:28] interesting pod/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 19:33:39.503784 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:39.503307 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 19:33:40.339644 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:40.339559 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:40.503407 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:40.503370 2582 patch_prober.go:28] interesting pod/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 19:33:40.504007 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:40.503960 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 19:33:40.708889 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:40.708747 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:40.802704 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:40.802603 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:41.506271 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:41.506226 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:41.506650 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:41.506506 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:41.507059 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:41.507045 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:33:44.789505 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:44.789474 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:44.890405 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:33:44.890373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:09.769433 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:34:09.769389 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:35:48.250206 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:35:48.250176 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:35:48.256370 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:35:48.256350 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jjrx_aa209ff4-1128-4cdb-afd6-aeafafcf0fd2/ovn-acl-logging/0.log" Jul 15 19:36:02.944704 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:02.944664 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:03.030168 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:03.030134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:16.823573 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:16.823535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:17.670097 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:17.670054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:23.200307 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:23.200268 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:31.014231 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:31.014191 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:32.659326 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:32.659292 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:48.778283 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:36:48.778234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:37:37.497950 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:37:37.497910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:37:38.389695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:37:38.389661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:37:52.445189 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:37:52.445152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:04.154303 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:04.154264 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:18.575276 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:18.575176 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:34.040100 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:34.040058 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:34.140766 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:34.140737 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:41.545133 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:41.545096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:41.643682 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:41.643642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:45.608125 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:45.608091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:45.703364 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:38:45.703329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:12.923774 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:12.923722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:13.693057 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:13.693022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:16.392297 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:16.392226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:20.590109 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.590075 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:20.592559 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.592542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:20.596318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.596294 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bd68r\"" Jul 15 19:39:20.600078 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.599855 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:20.660149 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.660114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxrp\" (UniqueName: \"kubernetes.io/projected/6d4bac86-c4f7-4e39-b24a-cf8535f82569-kube-api-access-9hxrp\") pod \"test-curl-7742-139723951193376\" (UID: \"6d4bac86-c4f7-4e39-b24a-cf8535f82569\") " pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:20.761415 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.761378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxrp\" (UniqueName: \"kubernetes.io/projected/6d4bac86-c4f7-4e39-b24a-cf8535f82569-kube-api-access-9hxrp\") pod \"test-curl-7742-139723951193376\" (UID: \"6d4bac86-c4f7-4e39-b24a-cf8535f82569\") " pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:20.771641 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.771611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxrp\" (UniqueName: \"kubernetes.io/projected/6d4bac86-c4f7-4e39-b24a-cf8535f82569-kube-api-access-9hxrp\") pod \"test-curl-7742-139723951193376\" (UID: \"6d4bac86-c4f7-4e39-b24a-cf8535f82569\") " pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:20.903435 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:20.903346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:21.046367 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:21.046342 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:21.047817 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:39:21.047788 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d4bac86_c4f7_4e39_b24a_cf8535f82569.slice/crio-7f02f28c62c4766798f6dfacd3e4b933ac9f7fecbbc3688ac6ce7f81844e90a7 WatchSource:0}: Error finding container 7f02f28c62c4766798f6dfacd3e4b933ac9f7fecbbc3688ac6ce7f81844e90a7: Status 404 returned error can't find the container with id 7f02f28c62c4766798f6dfacd3e4b933ac9f7fecbbc3688ac6ce7f81844e90a7 Jul 15 19:39:21.049126 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:21.049109 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:39:22.038041 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:22.038009 2582 generic.go:356] "Generic (PLEG): container finished" podID="6d4bac86-c4f7-4e39-b24a-cf8535f82569" containerID="c24ea9782be70f12cabcfe29119eec63516cece29f2935de0a9db9f418543084" exitCode=0 Jul 15 19:39:22.038453 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:22.038064 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723951193376" event={"ID":"6d4bac86-c4f7-4e39-b24a-cf8535f82569","Type":"ContainerDied","Data":"c24ea9782be70f12cabcfe29119eec63516cece29f2935de0a9db9f418543084"} Jul 15 19:39:22.038453 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:22.038113 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723951193376" event={"ID":"6d4bac86-c4f7-4e39-b24a-cf8535f82569","Type":"ContainerStarted","Data":"7f02f28c62c4766798f6dfacd3e4b933ac9f7fecbbc3688ac6ce7f81844e90a7"} Jul 15 19:39:23.179442 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:23.179418 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:23.207862 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:23.207830 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139723951193376_6d4bac86-c4f7-4e39-b24a-cf8535f82569/test-curl-7742-139723951193376/0.log" Jul 15 19:39:23.236308 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:23.236269 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:23.239728 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:23.239702 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:23.284904 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:23.284871 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hxrp\" (UniqueName: \"kubernetes.io/projected/6d4bac86-c4f7-4e39-b24a-cf8535f82569-kube-api-access-9hxrp\") pod \"6d4bac86-c4f7-4e39-b24a-cf8535f82569\" (UID: \"6d4bac86-c4f7-4e39-b24a-cf8535f82569\") " Jul 15 19:39:23.286980 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:23.286954 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4bac86-c4f7-4e39-b24a-cf8535f82569-kube-api-access-9hxrp" (OuterVolumeSpecName: "kube-api-access-9hxrp") pod "6d4bac86-c4f7-4e39-b24a-cf8535f82569" (UID: "6d4bac86-c4f7-4e39-b24a-cf8535f82569"). InnerVolumeSpecName "kube-api-access-9hxrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:23.386005 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:23.385920 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hxrp\" (UniqueName: \"kubernetes.io/projected/6d4bac86-c4f7-4e39-b24a-cf8535f82569-kube-api-access-9hxrp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:24.049834 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:24.049806 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:24.049834 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:24.049837 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f02f28c62c4766798f6dfacd3e4b933ac9f7fecbbc3688ac6ce7f81844e90a7" Jul 15 19:39:24.059759 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:39:24.059731 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7742-139723951193376\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="6d4bac86-c4f7-4e39-b24a-cf8535f82569" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:24.198789 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:24.198758 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d4bac86-c4f7-4e39-b24a-cf8535f82569" path="/var/lib/kubelet/pods/6d4bac86-c4f7-4e39-b24a-cf8535f82569/volumes" Jul 15 19:39:32.044303 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:32.044256 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:37.418728 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:37.418688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:38.422168 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:38.422127 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:38.518063 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:38.518023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:38.671231 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:38.671192 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl"] Jul 15 19:39:38.671497 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:38.671453 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" containerID="cri-o://b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d" gracePeriod=30 Jul 15 19:39:38.677213 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:38.677145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:38.908901 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:38.908863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.346133 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:39.346098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:41.878870 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:41.878833 2582 patch_prober.go:28] interesting pod/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:39:41.879254 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:41.878889 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:39:43.934012 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:43.933957 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:39:44.076684 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076656 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2z9b\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-kube-api-access-n2z9b\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.076873 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076725 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-socket\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.076873 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076753 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-data\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.076873 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076784 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istiod-ca-cert\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.076873 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076817 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-podinfo\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.076873 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076859 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-credential-socket\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.077147 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076883 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-envoy\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.077147 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076915 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-certs\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.077147 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.076939 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-token\") pod \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\" (UID: \"15f8ead7-e65c-4403-ac4a-63b69cb39ef4\") " Jul 15 19:39:44.077147 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077093 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:44.077386 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077193 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:44.077386 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077347 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:44.077582 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077551 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:39:44.077718 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077623 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.077718 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077640 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.077718 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077650 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.077896 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.077819 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-data" (OuterVolumeSpecName: "istio-data") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:44.079006 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.078982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-kube-api-access-n2z9b" (OuterVolumeSpecName: "kube-api-access-n2z9b") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "kube-api-access-n2z9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:44.079126 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.079074 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:44.079422 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.079398 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:39:44.079507 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.079447 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-token" (OuterVolumeSpecName: "istio-token") pod "15f8ead7-e65c-4403-ac4a-63b69cb39ef4" (UID: "15f8ead7-e65c-4403-ac4a-63b69cb39ef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:44.154167 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.154136 2582 generic.go:356] "Generic (PLEG): container finished" podID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerID="b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d" exitCode=0 Jul 15 19:39:44.154330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.154186 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" Jul 15 19:39:44.154330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.154223 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" event={"ID":"15f8ead7-e65c-4403-ac4a-63b69cb39ef4","Type":"ContainerDied","Data":"b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d"} Jul 15 19:39:44.154330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.154301 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl" event={"ID":"15f8ead7-e65c-4403-ac4a-63b69cb39ef4","Type":"ContainerDied","Data":"50cd0d95c17f297fcbdba288086e9676a3c099c2a5112633d20d3f26e992ce76"} Jul 15 19:39:44.154330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.154324 2582 scope.go:117] "RemoveContainer" containerID="b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d" Jul 15 19:39:44.163949 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.163932 2582 scope.go:117] "RemoveContainer" containerID="dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487" Jul 15 19:39:44.171558 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.171543 2582 scope.go:117] "RemoveContainer" containerID="b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d" Jul 15 19:39:44.171794 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:39:44.171775 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d\": container with ID starting with b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d not found: ID does not exist" containerID="b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d" Jul 15 19:39:44.171858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.171803 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d"} err="failed to get container status \"b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d\": rpc error: code = NotFound desc = could not find container \"b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d\": container with ID starting with b829cf879f94eafa6b98302ccb06c196b6972a8cb3da7ed35e155fbd1186e67d not found: ID does not exist" Jul 15 19:39:44.171858 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.171821 2582 scope.go:117] "RemoveContainer" containerID="dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487" Jul 15 19:39:44.172032 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:39:44.172014 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487\": container with ID starting with dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487 not found: ID does not exist" containerID="dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487" Jul 15 19:39:44.172070 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.172036 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487"} err="failed to get container status \"dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487\": rpc error: code = NotFound desc = could not find container \"dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487\": container with ID starting with dcab8304fc17bed8f5e55134ab176fd63457b3a917eae206f227a29885d9c487 not found: ID does not exist" Jul 15 19:39:44.178456 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.178437 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.178456 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.178457 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.178586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.178465 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.178586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.178475 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2z9b\" (UniqueName: \"kubernetes.io/projected/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-kube-api-access-n2z9b\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.178586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.178488 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.178586 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.178497 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f8ead7-e65c-4403-ac4a-63b69cb39ef4-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:44.178709 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.178642 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl"] Jul 15 19:39:44.184868 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.184812 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-4a4ed2-openshift-default-7bcdd85bf5-r5skl"] Jul 15 19:39:44.199156 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:44.199135 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" path="/var/lib/kubelet/pods/15f8ead7-e65c-4403-ac4a-63b69cb39ef4/volumes" Jul 15 19:39:45.025254 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:45.025200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:45.567937 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:45.567895 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:51.329892 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:51.329856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:51.428163 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:51.428124 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:51.582281 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:51.582188 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6"] Jul 15 19:39:51.582491 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:51.582460 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" containerID="cri-o://e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896" gracePeriod=30 Jul 15 19:39:51.605930 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:51.605904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:51.832676 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:51.832583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:52.263633 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:52.263607 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7999c8f867-kkgcp_46e5766e-2a33-4d5d-a45f-0cbbfa86325c/authorino/0.log" Jul 15 19:39:56.509011 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.508971 2582 patch_prober.go:28] interesting pod/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:39:56.509410 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.509029 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:39:56.837545 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.837519 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:39:56.984110 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984075 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-istio-token\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984118 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-certs\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984139 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77717dc8-3b93-4d11-bb74-d774af63f642-istio-podinfo\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984156 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-socket\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984189 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77717dc8-3b93-4d11-bb74-d774af63f642-istiod-ca-cert\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984215 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-credential-socket\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984276 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5htkr\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-kube-api-access-5htkr\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984318 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984309 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-data\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984349 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-envoy\") pod \"77717dc8-3b93-4d11-bb74-d774af63f642\" (UID: \"77717dc8-3b93-4d11-bb74-d774af63f642\") " Jul 15 19:39:56.984695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984397 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:56.984695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984635 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:56.984695 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984650 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:56.984915 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77717dc8-3b93-4d11-bb74-d774af63f642-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:39:56.984970 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.984923 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:56.985441 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.985420 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-data" (OuterVolumeSpecName: "istio-data") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:56.986419 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.986392 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77717dc8-3b93-4d11-bb74-d774af63f642-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:39:56.986538 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.986518 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:56.986795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.986771 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-kube-api-access-5htkr" (OuterVolumeSpecName: "kube-api-access-5htkr") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "kube-api-access-5htkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:56.986795 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:56.986786 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-istio-token" (OuterVolumeSpecName: "istio-token") pod "77717dc8-3b93-4d11-bb74-d774af63f642" (UID: "77717dc8-3b93-4d11-bb74-d774af63f642"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:57.085190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085106 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.085190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085145 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.085190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085155 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.085190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085163 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77717dc8-3b93-4d11-bb74-d774af63f642-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.085190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085171 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.085190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085181 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77717dc8-3b93-4d11-bb74-d774af63f642-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.085190 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085190 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77717dc8-3b93-4d11-bb74-d774af63f642-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.085531 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.085200 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5htkr\" (UniqueName: \"kubernetes.io/projected/77717dc8-3b93-4d11-bb74-d774af63f642-kube-api-access-5htkr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Jul 15 19:39:57.226147 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.226115 2582 generic.go:356] "Generic (PLEG): container finished" podID="77717dc8-3b93-4d11-bb74-d774af63f642" containerID="e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896" exitCode=0 Jul 15 19:39:57.226365 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.226166 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" Jul 15 19:39:57.226365 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.226208 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" event={"ID":"77717dc8-3b93-4d11-bb74-d774af63f642","Type":"ContainerDied","Data":"e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896"} Jul 15 19:39:57.226365 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.226287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6" event={"ID":"77717dc8-3b93-4d11-bb74-d774af63f642","Type":"ContainerDied","Data":"854fbd0ef2e595c7684f8fc63da2b9c4e62c2a66a85708e8090e39c3abb6d7b2"} Jul 15 19:39:57.226365 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.226318 2582 scope.go:117] "RemoveContainer" containerID="e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896" Jul 15 19:39:57.236356 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.236342 2582 scope.go:117] "RemoveContainer" containerID="2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14" Jul 15 19:39:57.244516 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.244499 2582 scope.go:117] "RemoveContainer" containerID="e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896" Jul 15 19:39:57.244773 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:39:57.244754 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896\": container with ID starting with e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896 not found: ID does not exist" containerID="e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896" Jul 15 19:39:57.244831 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.244781 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896"} err="failed to get container status \"e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896\": rpc error: code = NotFound desc = could not find container \"e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896\": container with ID starting with e4b1f292c4301e2a85667cd0942c98be0a319032f7d847f16d9e23e789d64896 not found: ID does not exist" Jul 15 19:39:57.244831 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.244798 2582 scope.go:117] "RemoveContainer" containerID="2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14" Jul 15 19:39:57.245053 ip-10-0-141-106 kubenswrapper[2582]: E0715 19:39:57.245034 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14\": container with ID starting with 2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14 not found: ID does not exist" containerID="2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14" Jul 15 19:39:57.245102 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.245058 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14"} err="failed to get container status \"2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14\": rpc error: code = NotFound desc = could not find container \"2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14\": container with ID starting with 2d15beea95ed66f5383da767cd0ddd3691749cb17c4c0ee2536ade57a66f1b14 not found: ID does not exist" Jul 15 19:39:57.248938 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.248913 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6"] Jul 15 19:39:57.253228 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.253207 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f83827-openshift-default-6958cc4f79-nxzf6"] Jul 15 19:39:57.507599 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.507571 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b2hqs_bd31d359-a5d0-401a-be62-3608adbe2438/manager/0.log" Jul 15 19:39:57.628198 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.628161 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pvtl4_38c6ac96-3354-47b9-ab1f-d430f23e1f25/manager/0.log" Jul 15 19:39:57.754302 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.754270 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d65cd49d-zsvwc_d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1/manager/0.log" Jul 15 19:39:57.866927 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.866850 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-slhht_1673566b-590c-429b-bb0f-d83027cd493f/server/0.log" Jul 15 19:39:57.986233 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:57.986203 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7gvhd_eefbafad-7ce8-4f41-bcb9-2223375f54f7/manager/0.log" Jul 15 19:39:58.094881 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:58.094852 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-6njpt_8779e7fe-5d58-4525-a675-f3a82e5996f5/copy-manifests/0.log" Jul 15 19:39:58.101688 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:58.101662 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-6njpt_8779e7fe-5d58-4525-a675-f3a82e5996f5/manager/0.log" Jul 15 19:39:58.200833 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:58.200748 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" path="/var/lib/kubelet/pods/77717dc8-3b93-4d11-bb74-d774af63f642/volumes" Jul 15 19:39:58.718991 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:58.718960 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7884b47f6b-jq4bd_ff216ef1-e553-40aa-8bc2-e96cb1117dcb/maas-api/0.log" Jul 15 19:39:59.055730 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:59.055651 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zhsqm_1897f6e9-366f-48f8-b863-f4549f4f52ec/postgres/0.log" Jul 15 19:39:59.837970 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:59.837933 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp_433b012f-7bc3-4e6e-ac9f-436899404f23/util/0.log" Jul 15 19:39:59.845196 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:59.845168 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp_433b012f-7bc3-4e6e-ac9f-436899404f23/pull/0.log" Jul 15 19:39:59.851350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:59.851329 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp_433b012f-7bc3-4e6e-ac9f-436899404f23/extract/0.log" Jul 15 19:39:59.966014 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:59.965976 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g_6ae4218b-87c0-488e-b2fc-1d3052adcc1d/extract/0.log" Jul 15 19:39:59.971550 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:59.971528 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g_6ae4218b-87c0-488e-b2fc-1d3052adcc1d/util/0.log" Jul 15 19:39:59.977391 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:39:59.977371 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g_6ae4218b-87c0-488e-b2fc-1d3052adcc1d/pull/0.log" Jul 15 19:40:00.086987 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.086954 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq_a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71/extract/0.log" Jul 15 19:40:00.092773 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.092701 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq_a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71/util/0.log" Jul 15 19:40:00.098969 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.098943 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq_a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71/pull/0.log" Jul 15 19:40:00.209967 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.209932 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np_5e57dcbb-ff75-49cd-b516-260e548ea9ab/util/0.log" Jul 15 19:40:00.216124 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.216097 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np_5e57dcbb-ff75-49cd-b516-260e548ea9ab/pull/0.log" Jul 15 19:40:00.222807 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.222785 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np_5e57dcbb-ff75-49cd-b516-260e548ea9ab/extract/0.log" Jul 15 19:40:00.339769 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.339729 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7999c8f867-kkgcp_46e5766e-2a33-4d5d-a45f-0cbbfa86325c/authorino/0.log" Jul 15 19:40:00.463942 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.463851 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7vqqt_4f0e0590-6b30-4b59-9c86-adb58c91719b/manager/0.log" Jul 15 19:40:00.571291 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.571248 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vk66k_b570ac71-b47e-4404-ba69-f567f7e7349e/manager/0.log" Jul 15 19:40:00.801428 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:00.801394 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-g4tzg_e51f9177-f90e-4920-8fee-8c651f54ddbf/registry-server/0.log" Jul 15 19:40:01.163060 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:01.162967 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q94cz_b0833af5-0a67-496f-91cb-8b37d10f9c7f/manager/0.log" Jul 15 19:40:01.523611 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:01.523585 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j9hcl_73198a57-057f-4510-8d27-8c9349325bc9/istio-proxy/0.log" Jul 15 19:40:01.881532 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:01.881437 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-x6ccn_36114d69-e0aa-4dc6-b314-0ed3c1214b69/kube-auth-proxy/0.log" Jul 15 19:40:02.000860 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:02.000832 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-79zpb_e67fea28-4b13-4985-9a30-b34c5460e8c8/istio-proxy/1.log" Jul 15 19:40:02.134364 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:02.134289 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-nhxrx_eb4f397e-cd77-4591-8885-db46cc7b0d0e/payload-pre-processing/0.log" Jul 15 19:40:02.686128 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:02.686079 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f566w_57c97fa1-8ffd-414b-afff-a2a9eb725dbd/main/0.log" Jul 15 19:40:03.150330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:03.150298 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-stxcc_99896c41-20e5-4929-aece-3d5cf8a1a2cc/main/0.log" Jul 15 19:40:08.742844 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.742810 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-csbwp/must-gather-vtjh2"] Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743173 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743184 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743191 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743197 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743206 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743211 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743221 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d4bac86-c4f7-4e39-b24a-cf8535f82569" containerName="test-curl-7742-139723951193376" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743227 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d4bac86-c4f7-4e39-b24a-cf8535f82569" containerName="test-curl-7742-139723951193376" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743317 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743326 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="77717dc8-3b93-4d11-bb74-d774af63f642" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743334 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d4bac86-c4f7-4e39-b24a-cf8535f82569" containerName="test-curl-7742-139723951193376" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743341 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" Jul 15 19:40:08.743350 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743346 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" Jul 15 19:40:08.743748 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743395 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" Jul 15 19:40:08.743748 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.743402 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f8ead7-e65c-4403-ac4a-63b69cb39ef4" containerName="istio-proxy" Jul 15 19:40:08.747851 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.747834 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:08.752152 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.752131 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-csbwp\"/\"kube-root-ca.crt\"" Jul 15 19:40:08.752599 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.752581 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-csbwp\"/\"openshift-service-ca.crt\"" Jul 15 19:40:08.772371 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.772347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csbwp/must-gather-vtjh2"] Jul 15 19:40:08.783477 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.783454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39a3a1f3-804e-4b32-b526-b9db6029a08a-must-gather-output\") pod \"must-gather-vtjh2\" (UID: \"39a3a1f3-804e-4b32-b526-b9db6029a08a\") " pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:08.783577 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.783509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trntf\" (UniqueName: \"kubernetes.io/projected/39a3a1f3-804e-4b32-b526-b9db6029a08a-kube-api-access-trntf\") pod \"must-gather-vtjh2\" (UID: \"39a3a1f3-804e-4b32-b526-b9db6029a08a\") " pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:08.817780 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.817755 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:40:08.884976 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.884952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39a3a1f3-804e-4b32-b526-b9db6029a08a-must-gather-output\") pod \"must-gather-vtjh2\" (UID: \"39a3a1f3-804e-4b32-b526-b9db6029a08a\") " pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:08.885087 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.885001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trntf\" (UniqueName: \"kubernetes.io/projected/39a3a1f3-804e-4b32-b526-b9db6029a08a-kube-api-access-trntf\") pod \"must-gather-vtjh2\" (UID: \"39a3a1f3-804e-4b32-b526-b9db6029a08a\") " pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:08.885361 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.885345 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39a3a1f3-804e-4b32-b526-b9db6029a08a-must-gather-output\") pod \"must-gather-vtjh2\" (UID: \"39a3a1f3-804e-4b32-b526-b9db6029a08a\") " pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:08.894500 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.894474 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trntf\" (UniqueName: \"kubernetes.io/projected/39a3a1f3-804e-4b32-b526-b9db6029a08a-kube-api-access-trntf\") pod \"must-gather-vtjh2\" (UID: \"39a3a1f3-804e-4b32-b526-b9db6029a08a\") " pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:08.913482 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:08.913459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" limit=1024 Jul 15 19:40:09.057079 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:09.057004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csbwp/must-gather-vtjh2" Jul 15 19:40:09.215003 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:09.214979 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csbwp/must-gather-vtjh2"] Jul 15 19:40:09.217193 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:40:09.217163 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39a3a1f3_804e_4b32_b526_b9db6029a08a.slice/crio-57a2ba8c0f8c5958d8a327f78e01f49678c3fcf73b6636cc480bd1d5abf751d6 WatchSource:0}: Error finding container 57a2ba8c0f8c5958d8a327f78e01f49678c3fcf73b6636cc480bd1d5abf751d6: Status 404 returned error can't find the container with id 57a2ba8c0f8c5958d8a327f78e01f49678c3fcf73b6636cc480bd1d5abf751d6 Jul 15 19:40:09.289923 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:09.289892 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csbwp/must-gather-vtjh2" event={"ID":"39a3a1f3-804e-4b32-b526-b9db6029a08a","Type":"ContainerStarted","Data":"57a2ba8c0f8c5958d8a327f78e01f49678c3fcf73b6636cc480bd1d5abf751d6"} Jul 15 19:40:11.304907 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:11.304865 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csbwp/must-gather-vtjh2" event={"ID":"39a3a1f3-804e-4b32-b526-b9db6029a08a","Type":"ContainerStarted","Data":"5912f94923e777817159cf388543fdaf32de5e3b22b4424371a73991b25515ef"} Jul 15 19:40:11.305419 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:11.304921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csbwp/must-gather-vtjh2" event={"ID":"39a3a1f3-804e-4b32-b526-b9db6029a08a","Type":"ContainerStarted","Data":"093e695bfea863a0d91227db131b9bcf64c7dcf826ae752a1fd9b798e84994a9"} Jul 15 19:40:11.323101 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:11.323035 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-csbwp/must-gather-vtjh2" podStartSLOduration=2.250707007 podStartE2EDuration="3.323012444s" totalImagesPullingTime="1.072305437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:40:11.321091687 +0000 UTC m=+2363.716447658" watchObservedRunningTime="2026-07-15 19:40:11.323012444 +0000 UTC m=+2363.718368416" Jul 15 19:40:11.837073 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:11.837035 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-67swt_1d7aee3e-b9b2-46d9-ad33-15bf10f3bb34/global-pull-secret-syncer/0.log" Jul 15 19:40:11.974553 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:11.974523 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4v4sv_4465b344-e63b-4844-9941-04810dbc9814/konnectivity-agent/0.log" Jul 15 19:40:12.111502 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:12.111418 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-106.ec2.internal_98572c5ed91da58bf583a8cefb49b9cd/haproxy/0.log" Jul 15 19:40:15.966790 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:15.966749 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp_433b012f-7bc3-4e6e-ac9f-436899404f23/extract/0.log" Jul 15 19:40:15.992708 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:15.992675 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp_433b012f-7bc3-4e6e-ac9f-436899404f23/util/0.log" Jul 15 19:40:16.033205 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.033168 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599ggnp_433b012f-7bc3-4e6e-ac9f-436899404f23/pull/0.log" Jul 15 19:40:16.090390 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.090357 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g_6ae4218b-87c0-488e-b2fc-1d3052adcc1d/extract/0.log" Jul 15 19:40:16.145446 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.145406 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g_6ae4218b-87c0-488e-b2fc-1d3052adcc1d/util/0.log" Jul 15 19:40:16.182503 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.182469 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08qb6g_6ae4218b-87c0-488e-b2fc-1d3052adcc1d/pull/0.log" Jul 15 19:40:16.226568 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.226537 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq_a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71/extract/0.log" Jul 15 19:40:16.253167 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.253131 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq_a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71/util/0.log" Jul 15 19:40:16.285522 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.285490 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738njsq_a9c2907f-e2d3-4bd4-82c4-a22bcd6a7a71/pull/0.log" Jul 15 19:40:16.324352 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.324313 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np_5e57dcbb-ff75-49cd-b516-260e548ea9ab/extract/0.log" Jul 15 19:40:16.359387 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.359356 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np_5e57dcbb-ff75-49cd-b516-260e548ea9ab/util/0.log" Jul 15 19:40:16.396744 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.396711 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hx7np_5e57dcbb-ff75-49cd-b516-260e548ea9ab/pull/0.log" Jul 15 19:40:16.889687 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.889651 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7999c8f867-kkgcp_46e5766e-2a33-4d5d-a45f-0cbbfa86325c/authorino/0.log" Jul 15 19:40:16.948079 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.948047 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7vqqt_4f0e0590-6b30-4b59-9c86-adb58c91719b/manager/0.log" Jul 15 19:40:16.979891 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:16.979864 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vk66k_b570ac71-b47e-4404-ba69-f567f7e7349e/manager/0.log" Jul 15 19:40:17.068469 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:17.068433 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-g4tzg_e51f9177-f90e-4920-8fee-8c651f54ddbf/registry-server/0.log" Jul 15 19:40:17.361262 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:17.360634 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q94cz_b0833af5-0a67-496f-91cb-8b37d10f9c7f/manager/0.log" Jul 15 19:40:19.424690 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:19.424656 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bfxw_02f7be89-1806-4d3f-b535-81c3d89ca460/node-exporter/0.log" Jul 15 19:40:19.452216 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:19.452177 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bfxw_02f7be89-1806-4d3f-b535-81c3d89ca460/kube-rbac-proxy/0.log" Jul 15 19:40:19.477464 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:19.477430 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bfxw_02f7be89-1806-4d3f-b535-81c3d89ca460/init-textfile/0.log" Jul 15 19:40:21.443698 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.443664 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-7fmfw_c8b8f985-f6a5-4b0b-9588-663c9aab1550/networking-console-plugin/0.log" Jul 15 19:40:21.484110 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.484074 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt"] Jul 15 19:40:21.492524 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.492494 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.495793 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.495709 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-csbwp\"/\"default-dockercfg-cx64j\"" Jul 15 19:40:21.501573 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.501543 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt"] Jul 15 19:40:21.603728 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.603694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-sys\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.604029 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.604006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-lib-modules\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.604166 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.604149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dr5x\" (UniqueName: \"kubernetes.io/projected/2d688c6c-9d7d-43f1-a591-6ed71563258a-kube-api-access-5dr5x\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.604380 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.604350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-proc\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.604472 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.604423 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-podres\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.705964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.705875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-lib-modules\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.705964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.705920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dr5x\" (UniqueName: \"kubernetes.io/projected/2d688c6c-9d7d-43f1-a591-6ed71563258a-kube-api-access-5dr5x\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.705964 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.705961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-proc\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.706223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.705991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-podres\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.706223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.706029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-sys\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.706223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.706085 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-lib-modules\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.706223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.706112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-proc\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.706223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.706127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-sys\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.706223 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.706163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d688c6c-9d7d-43f1-a591-6ed71563258a-podres\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.714939 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.714913 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dr5x\" (UniqueName: \"kubernetes.io/projected/2d688c6c-9d7d-43f1-a591-6ed71563258a-kube-api-access-5dr5x\") pod \"perf-node-gather-daemonset-f2mpt\" (UID: \"2d688c6c-9d7d-43f1-a591-6ed71563258a\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:21.812339 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:21.812309 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:22.007529 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:22.007497 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt"] Jul 15 19:40:22.008723 ip-10-0-141-106 kubenswrapper[2582]: W0715 19:40:22.008690 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2d688c6c_9d7d_43f1_a591_6ed71563258a.slice/crio-00d2c1787dfc07640909616441a80b9eec12ef3ecbe00fd0afe77cbefe142022 WatchSource:0}: Error finding container 00d2c1787dfc07640909616441a80b9eec12ef3ecbe00fd0afe77cbefe142022: Status 404 returned error can't find the container with id 00d2c1787dfc07640909616441a80b9eec12ef3ecbe00fd0afe77cbefe142022 Jul 15 19:40:22.399356 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:22.399266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" event={"ID":"2d688c6c-9d7d-43f1-a591-6ed71563258a","Type":"ContainerStarted","Data":"62ec16bbf92489deef0e508a89d96ad318af8ae5c54d9444da0e71e6d24f7f63"} Jul 15 19:40:22.399622 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:22.399599 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" event={"ID":"2d688c6c-9d7d-43f1-a591-6ed71563258a","Type":"ContainerStarted","Data":"00d2c1787dfc07640909616441a80b9eec12ef3ecbe00fd0afe77cbefe142022"} Jul 15 19:40:22.399950 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:22.399927 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:22.417609 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:22.417549 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" podStartSLOduration=1.4175281960000001 podStartE2EDuration="1.417528196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:40:22.415769087 +0000 UTC m=+2374.811125058" watchObservedRunningTime="2026-07-15 19:40:22.417528196 +0000 UTC m=+2374.812884166" Jul 15 19:40:24.449197 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:24.449167 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mw954_36635b29-e242-40ea-89c8-75dffb3c7324/dns/0.log" Jul 15 19:40:24.473483 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:24.473448 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mw954_36635b29-e242-40ea-89c8-75dffb3c7324/kube-rbac-proxy/0.log" Jul 15 19:40:24.608148 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:24.608116 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pc459_3aace167-d7c8-4990-af41-4318ad2a5d28/dns-node-resolver/0.log" Jul 15 19:40:25.186863 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:25.186827 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-94b8f45fb-n2shh_c40cce77-bb87-40d5-82b0-b8c7708ec552/registry/0.log" Jul 15 19:40:25.238706 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:25.238674 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hw4qx_b3bae5c4-d541-4f2f-aaea-e830abbe5212/node-ca/0.log" Jul 15 19:40:26.239010 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:26.238982 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j9hcl_73198a57-057f-4510-8d27-8c9349325bc9/istio-proxy/0.log" Jul 15 19:40:26.394727 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:26.394683 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-x6ccn_36114d69-e0aa-4dc6-b314-0ed3c1214b69/kube-auth-proxy/0.log" Jul 15 19:40:26.430048 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:26.429991 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-79zpb_e67fea28-4b13-4985-9a30-b34c5460e8c8/istio-proxy/0.log" Jul 15 19:40:26.496096 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:26.496005 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-79zpb_e67fea28-4b13-4985-9a30-b34c5460e8c8/istio-proxy/1.log" Jul 15 19:40:26.535120 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:26.535088 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-nhxrx_eb4f397e-cd77-4591-8885-db46cc7b0d0e/payload-pre-processing/0.log" Jul 15 19:40:27.275590 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:27.275559 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bljkh_f62bae5e-1f0d-4d93-92a3-e272e8d9353f/serve-healthcheck-canary/0.log" Jul 15 19:40:27.940956 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:27.940923 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8shls_58ff17fe-ed94-43f3-9798-816ae901a86f/kube-rbac-proxy/0.log" Jul 15 19:40:27.971275 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:27.971225 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8shls_58ff17fe-ed94-43f3-9798-816ae901a86f/exporter/0.log" Jul 15 19:40:28.007425 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:28.007385 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8shls_58ff17fe-ed94-43f3-9798-816ae901a86f/extractor/0.log" Jul 15 19:40:28.420365 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:28.420332 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-f2mpt" Jul 15 19:40:30.865983 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:30.865955 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b2hqs_bd31d359-a5d0-401a-be62-3608adbe2438/manager/0.log" Jul 15 19:40:31.106315 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:31.105422 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pvtl4_38c6ac96-3354-47b9-ab1f-d430f23e1f25/manager/0.log" Jul 15 19:40:31.281643 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:31.281486 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d65cd49d-zsvwc_d1d4fa3b-c102-495f-b0e8-7fc80e38e8d1/manager/0.log" Jul 15 19:40:31.306632 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:31.306604 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-slhht_1673566b-590c-429b-bb0f-d83027cd493f/server/0.log" Jul 15 19:40:31.414330 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:31.414295 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7gvhd_eefbafad-7ce8-4f41-bcb9-2223375f54f7/manager/0.log" Jul 15 19:40:31.445279 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:31.445233 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-6njpt_8779e7fe-5d58-4525-a675-f3a82e5996f5/manager/0.log" Jul 15 19:40:31.469857 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:31.469830 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-6njpt_8779e7fe-5d58-4525-a675-f3a82e5996f5/copy-manifests/0.log" Jul 15 19:40:32.630217 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:32.630180 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j9hcl_73198a57-057f-4510-8d27-8c9349325bc9/istio-proxy/0.log" Jul 15 19:40:32.757217 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:32.757185 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-x6ccn_36114d69-e0aa-4dc6-b314-0ed3c1214b69/kube-auth-proxy/0.log" Jul 15 19:40:32.782983 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:32.782952 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-79zpb_e67fea28-4b13-4985-9a30-b34c5460e8c8/istio-proxy/0.log" Jul 15 19:40:32.854070 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:32.853928 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-79zpb_e67fea28-4b13-4985-9a30-b34c5460e8c8/istio-proxy/1.log" Jul 15 19:40:32.878821 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:32.878786 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-nhxrx_eb4f397e-cd77-4591-8885-db46cc7b0d0e/payload-pre-processing/0.log" Jul 15 19:40:33.007109 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:33.007074 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-75z8g_c836347c-92d2-415d-bfd5-26087755e779/openshift-lws-operator/0.log" Jul 15 19:40:34.153417 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:34.153372 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j9hcl_73198a57-057f-4510-8d27-8c9349325bc9/istio-proxy/0.log" Jul 15 19:40:34.304189 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:34.304163 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-x6ccn_36114d69-e0aa-4dc6-b314-0ed3c1214b69/kube-auth-proxy/0.log" Jul 15 19:40:34.331058 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:34.331031 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-79zpb_e67fea28-4b13-4985-9a30-b34c5460e8c8/istio-proxy/0.log" Jul 15 19:40:34.399097 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:34.399071 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-79zpb_e67fea28-4b13-4985-9a30-b34c5460e8c8/istio-proxy/1.log" Jul 15 19:40:34.421632 ip-10-0-141-106 kubenswrapper[2582]: I0715 19:40:34.421551 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-nhxrx_eb4f397e-cd77-4591-8885-db46cc7b0d0e/payload-pre-processing/0.log"