Jul 13 14:46:02.208457 ip-10-0-133-41 systemd[1]: Starting Kubernetes Kubelet... Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.668928 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676346 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676360 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676364 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676367 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676370 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676375 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676378 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:46:02.676884 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676380 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676383 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676385 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676388 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676391 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676394 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676398 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676401 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676404 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676407 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676409 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676414 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676418 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676421 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676424 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676427 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676430 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676432 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676435 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:46:02.691940 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676438 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676441 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676444 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676448 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676450 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676453 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676457 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676459 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676462 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676465 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676467 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676470 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676473 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676476 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676478 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676480 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676483 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676486 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676489 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:46:02.693692 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676491 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676494 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676497 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676500 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676503 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676506 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676508 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676511 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676513 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676515 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676518 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676520 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676523 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676525 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676528 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676531 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676534 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676538 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676541 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:46:02.695862 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676544 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676547 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676550 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676552 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676555 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676557 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676560 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676563 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676566 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676568 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676570 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676573 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676575 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676578 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676581 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676583 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676586 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676588 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676591 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676594 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:46:02.711448 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676596 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676599 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676601 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676604 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676607 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676609 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676612 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676614 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676616 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676619 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676622 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676624 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676627 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676630 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.676633 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677360 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677368 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677371 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677374 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677378 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:46:02.713855 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677382 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677385 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677387 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677390 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677393 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677395 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677398 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677401 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677403 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677406 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677409 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677412 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677416 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677419 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677421 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677424 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677426 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677429 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677431 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677434 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:46:02.715664 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677436 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677439 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677442 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677445 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677447 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677450 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677453 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677456 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677458 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677461 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677463 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677466 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677469 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677472 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677475 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677477 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677480 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677483 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677485 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677488 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:46:02.718137 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677490 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677493 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677495 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677498 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677501 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677504 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677508 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677512 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677515 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677518 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677521 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677524 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677527 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677532 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677534 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677537 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677539 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677542 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677544 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:46:02.720090 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677547 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677550 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677553 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677555 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677558 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677561 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677564 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677566 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677569 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677571 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677574 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677576 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677579 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677581 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677586 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677589 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677592 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677595 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677598 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677601 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:46:02.722540 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677603 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677606 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677608 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677611 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677615 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677618 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677622 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677625 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677628 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677631 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677634 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677636 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677639 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677641 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.677644 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678010 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678016 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678018 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678021 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:46:02.792878 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678023 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678026 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678029 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678032 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678034 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678037 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678040 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678043 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678045 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678048 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678053 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678056 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678059 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678062 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678065 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678069 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678072 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678074 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678077 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678082 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:46:02.853553 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678085 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:46:02.748458 ip-10-0-133-41 systemd[1]: Started Kubernetes Kubelet. Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678087 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678090 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678093 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678096 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678098 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678100 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678103 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678105 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678108 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678110 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678114 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678118 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678121 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678123 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678126 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678129 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678132 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678134 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:46:03.020396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678137 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678140 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678143 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678146 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678171 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678175 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678178 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678181 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678184 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678186 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678189 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678191 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678196 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678198 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678200 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678203 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678206 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678208 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678211 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678213 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:46:03.021112 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678215 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678218 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678221 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678223 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678226 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678228 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678230 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678233 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678235 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678239 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678242 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678244 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678247 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678250 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678254 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678258 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678261 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678263 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678266 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678269 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:46:03.022823 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678272 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678275 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678277 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678280 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678283 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678286 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678288 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678291 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678294 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678296 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678299 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678301 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678304 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678306 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678309 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.678311 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678393 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678403 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678413 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678419 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678426 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 14:46:03.024191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678430 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678434 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678438 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678441 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678444 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678447 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678452 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678456 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678458 2571 flags.go:64] FLAG: --cgroup-root="" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678461 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678464 2571 flags.go:64] FLAG: --client-ca-file="" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678470 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678474 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678477 2571 flags.go:64] FLAG: --cluster-domain="" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678480 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678485 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678488 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678491 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678494 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678497 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678500 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678503 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678507 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678510 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 14:46:03.026436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678512 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678516 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678518 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678523 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678526 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678529 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678531 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678534 2571 flags.go:64] FLAG: --enable-server="true" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678537 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678541 2571 flags.go:64] FLAG: --event-burst="100" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678544 2571 flags.go:64] FLAG: --event-qps="50" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678547 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678550 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678553 2571 flags.go:64] FLAG: --eviction-hard="" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678558 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678561 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678564 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678567 2571 flags.go:64] FLAG: --eviction-soft="" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678570 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678573 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678576 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678579 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678582 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678586 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678589 2571 flags.go:64] FLAG: --feature-gates="" Jul 13 14:46:03.028338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678592 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678595 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678598 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678601 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678604 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678607 2571 flags.go:64] FLAG: --help="false" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678610 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678613 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678616 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678618 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678622 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678625 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678628 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678631 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678633 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678636 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678639 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678642 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678645 2571 flags.go:64] FLAG: --kube-reserved="" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678647 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678650 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678654 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678657 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678660 2571 flags.go:64] FLAG: --lock-file="" Jul 13 14:46:03.030225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678663 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678665 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678668 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678674 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678677 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678680 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678684 2571 flags.go:64] FLAG: --logging-format="text" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678687 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678690 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678693 2571 flags.go:64] FLAG: --manifest-url="" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678696 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678700 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678703 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678706 2571 flags.go:64] FLAG: --max-pods="110" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678709 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678712 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678715 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678718 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678721 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678723 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678726 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678733 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678736 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678739 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 14:46:03.031335 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678742 2571 flags.go:64] FLAG: --pod-cidr="" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678745 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678750 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678753 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678756 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678759 2571 flags.go:64] FLAG: --port="10250" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678763 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678766 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0012d42163dda2a40" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678770 2571 flags.go:64] FLAG: --qos-reserved="" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678773 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678776 2571 flags.go:64] FLAG: --register-node="true" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678778 2571 flags.go:64] FLAG: --register-with-taints="" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678787 2571 flags.go:64] FLAG: --registry-burst="10" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678790 2571 flags.go:64] FLAG: --registry-qps="5" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678795 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678797 2571 flags.go:64] FLAG: --reserved-memory="" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678801 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678804 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678807 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678811 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678813 2571 flags.go:64] FLAG: --runonce="false" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678816 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678819 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678822 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678825 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 14:46:03.032984 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678828 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678831 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678834 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678837 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678840 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678842 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678845 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678848 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678851 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678854 2571 flags.go:64] FLAG: --system-cgroups="" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678856 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678861 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678864 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678868 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678872 2571 flags.go:64] FLAG: --tls-min-version="" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678875 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678878 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678880 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678883 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678886 2571 flags.go:64] FLAG: --v="2" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678890 2571 flags.go:64] FLAG: --version="false" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678897 2571 flags.go:64] FLAG: --vmodule="" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678900 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.678903 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679014 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:46:03.034716 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679019 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679022 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679026 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679029 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679031 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679034 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679037 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679040 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679042 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679045 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679049 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679053 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679056 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679058 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679061 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679064 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679067 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679071 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679074 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:46:03.035852 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679077 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679082 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679085 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679089 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679091 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679095 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679098 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679100 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679103 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679107 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679110 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679113 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679116 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679119 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679121 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679124 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679127 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679129 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679132 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:46:03.036644 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679134 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679137 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679139 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679142 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679144 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679147 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679166 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679168 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679171 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679173 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679176 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679179 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679181 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679184 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679188 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679191 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679193 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679196 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679198 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679201 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:46:03.037458 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679204 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679206 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679210 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679212 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679215 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679218 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679221 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679223 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679226 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679229 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679231 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679234 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679236 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679238 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679241 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679244 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679246 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679249 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679251 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679254 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:46:03.038488 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679257 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679261 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679264 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679266 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679269 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679272 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679276 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679279 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679281 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679284 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679286 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679289 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679292 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679294 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679298 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679301 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679303 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679306 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679309 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:46:03.039229 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.679311 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.680172 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.687601 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.687617 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.687636 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.687646 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687700 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687704 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687708 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687711 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687714 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687716 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687720 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687722 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687725 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:46:03.039807 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687728 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687731 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687734 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687736 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687739 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687742 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687745 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687747 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687750 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687752 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687757 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687760 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687763 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687766 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687769 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687772 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687775 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687777 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687780 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:46:03.040256 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687782 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687785 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687787 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687790 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687792 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687795 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687797 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687800 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687803 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687805 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687808 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687810 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687813 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687816 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687818 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687823 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687826 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687828 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687831 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687834 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:46:03.040805 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687836 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687839 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687842 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687845 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687847 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687850 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687852 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687855 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687859 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687862 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687864 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687867 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687869 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687872 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687875 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687877 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687880 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687882 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687885 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687887 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:46:03.041370 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687890 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687892 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687895 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687897 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687900 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687903 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687905 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687909 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687912 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687915 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687917 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687920 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687924 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687928 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687931 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687934 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687937 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687940 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687942 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:46:03.041917 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687945 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687951 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687954 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687957 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687960 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687963 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687966 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687968 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687971 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687974 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687976 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.687979 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.687984 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688083 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688087 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:46:03.042752 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688090 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688093 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688096 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688099 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688101 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688105 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688109 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688112 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688114 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688117 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688119 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688122 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688124 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688127 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688130 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688132 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688136 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688138 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688142 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688145 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:46:03.234286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688165 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688170 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688172 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688175 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688177 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688181 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688185 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688188 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688190 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688193 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688196 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688199 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688201 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688204 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688206 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688208 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688212 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688216 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688219 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:46:03.235679 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688222 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688224 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688227 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688230 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688233 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688235 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688238 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688240 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688243 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688245 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688248 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688252 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688254 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688257 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688259 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688262 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688265 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688267 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688270 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:46:03.236289 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688272 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688275 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688277 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688280 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688282 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688285 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688287 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688290 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688292 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688295 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688298 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688300 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688303 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688306 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688308 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688312 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688316 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688319 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688321 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:46:03.236802 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688324 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688327 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688330 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688333 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688335 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688339 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688342 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688345 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688347 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688349 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688353 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688355 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688358 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688374 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688379 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688382 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688385 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688388 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688391 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:46:03.237389 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:02.688393 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.688398 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.688499 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.691783 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.693603 2571 server.go:1019] "Starting client certificate rotation" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.693710 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.693740 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.696788 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.697904 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.710115 2571 log.go:25] "Validated CRI v1 runtime API" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.713668 2571 log.go:25] "Validated CRI v1 image API" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.715857 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 14:46:03.237906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.719199 2571 fs.go:135] Filesystem UUIDs: map[06c23a5b-7003-4b3f-a61a-174f2b68d498:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9cf76aef-1b0f-4424-918a-f587103b8ada:/dev/nvme0n1p4] Jul 13 14:46:03.238343 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.719230 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.725580 2571 manager.go:216] Machine: {Timestamp:2026-07-13 14:46:02.724170483 +0000 UTC m=+0.378136929 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099493 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d3682edfb5353aaafb5b471283344 SystemUUID:ec2d3682-edfb-5353-aaaf-b5b471283344 BootID:4566b988-ed13-4b51-939f-36b88ec859b5 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2a:f1:b2:fb:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2a:f1:b2:fb:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:8b:c9:0f:22:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.725679 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.725785 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.729109 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.729134 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-41.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.730072 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.730086 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 14:46:03.238427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.730094 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.730107 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.731420 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736307 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736418 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736441 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736443 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736459 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736476 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736485 2571 kubelet.go:398] "Adding apiserver pod source" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736494 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736788 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.736802 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.739284 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741087 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741114 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741932 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741945 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741950 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741956 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741963 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.741969 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.742061 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.742067 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.742074 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.742079 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.742088 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.742096 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.742998 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.743008 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.747630 2571 server.go:1268] "Started kubelet" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.747763 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.747826 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.748626 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.749352 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.751534 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.752386 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.752391 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.752392 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-41.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 14:46:03.238746 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.754908 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.755504 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.755516 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.755616 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756280 2571 factory.go:55] Registering systemd factory Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756295 2571 factory.go:223] Registration of the systemd container factory successfully Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756318 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756318 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756335 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756415 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756423 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756546 2571 factory.go:153] Registering CRI-O factory Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756558 2571 factory.go:223] Registration of the crio container factory successfully Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756608 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756632 2571 factory.go:103] Registering Raw factory Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.756660 2571 manager.go:1196] Started watching for new ooms in manager Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.757043 2571 manager.go:349] Starting recovery of all containers Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.756368 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-41.ec2.internal.18c1e0ad653775a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-41.ec2.internal,UID:ip-10-0-133-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-41.ec2.internal,},FirstTimestamp:2026-07-13 14:46:02.747524515 +0000 UTC m=+0.401490961,LastTimestamp:2026-07-13 14:46:02.747524515 +0000 UTC m=+0.401490961,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-41.ec2.internal,}" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.764676 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58mml" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.765766 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.765796 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.766270 2571 manager.go:354] Recovery completed Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.770622 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.772847 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.772872 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.772883 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.773367 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.773379 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.773396 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.773380 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.773626 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58mml" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.774056 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:46:03.239779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.774077 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.774086 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.774916 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-41.ec2.internal.18c1e0ad66ba042c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-41.ec2.internal,UID:ip-10-0-133-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-41.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-41.ec2.internal,},FirstTimestamp:2026-07-13 14:46:02.7728579 +0000 UTC m=+0.426824345,LastTimestamp:2026-07-13 14:46:02.7728579 +0000 UTC m=+0.426824345,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-41.ec2.internal,}" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.775540 2571 policy_none.go:49] "None policy: Start" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.775555 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.775569 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.777629 2571 policy_none.go:47] "Start" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.856432 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.862223 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.863441 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.863481 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.863508 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.863548 2571 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.866328 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.878679 2571 manager.go:337] "Starting Device Plugin manager" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.878708 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.878717 2571 server.go:87] "Starting device plugin registration server" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.878951 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.879081 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.879198 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.879291 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.879300 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.879727 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.879776 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.964526 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal"] Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.964609 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.965720 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.965750 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.965761 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.967977 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968107 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968138 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:46:03.240684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968725 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968748 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968771 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968780 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968752 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.968838 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.971000 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.971026 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.971827 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.971842 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.971856 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.975319 2571 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-41.ec2.internal\" not found" node="ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.979481 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.980098 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.980122 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.980134 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.980170 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:02.989641 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:02.989657 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-41.ec2.internal\": node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.005122 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.013910 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-41.ec2.internal\" not found" node="ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.019889 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-41.ec2.internal\" not found" node="ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.057496 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be6b691ccebfc90f351c1aa81ac9e62f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal\" (UID: \"be6b691ccebfc90f351c1aa81ac9e62f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.057521 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102e87a82f37db84cece8bdb72d68ea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-41.ec2.internal\" (UID: \"102e87a82f37db84cece8bdb72d68ea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.057538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be6b691ccebfc90f351c1aa81ac9e62f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal\" (UID: \"be6b691ccebfc90f351c1aa81ac9e62f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.241548 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.105624 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.242390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.157966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be6b691ccebfc90f351c1aa81ac9e62f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal\" (UID: \"be6b691ccebfc90f351c1aa81ac9e62f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.242390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.157996 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be6b691ccebfc90f351c1aa81ac9e62f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal\" (UID: \"be6b691ccebfc90f351c1aa81ac9e62f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.242390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.158016 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102e87a82f37db84cece8bdb72d68ea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-41.ec2.internal\" (UID: \"102e87a82f37db84cece8bdb72d68ea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.242390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.158057 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102e87a82f37db84cece8bdb72d68ea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-41.ec2.internal\" (UID: \"102e87a82f37db84cece8bdb72d68ea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.242390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.158073 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be6b691ccebfc90f351c1aa81ac9e62f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal\" (UID: \"be6b691ccebfc90f351c1aa81ac9e62f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.242390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.158090 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be6b691ccebfc90f351c1aa81ac9e62f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal\" (UID: \"be6b691ccebfc90f351c1aa81ac9e62f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.242390 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.206200 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.433692 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.306858 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.433692 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.317020 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.433692 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.323145 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" Jul 13 14:46:03.433692 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.407424 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.654050 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.507892 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.654050 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.611484 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.693209 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.693366 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.712519 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.736693 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.779432 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 14:41:03 +0000 UTC" deadline="2028-04-28 12:35:10.009925962 +0000 UTC" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.779462 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15717h49m6.230467024s" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:03.812696 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-41.ec2.internal\" not found" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.857098 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:46:03.928129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.881281 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:46:04.192851 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.955812 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" Jul 13 14:46:04.192851 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.968606 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 14:46:04.192851 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.969505 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" Jul 13 14:46:04.192851 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:03.978333 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 14:46:04.192851 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:04.037130 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6b691ccebfc90f351c1aa81ac9e62f.slice/crio-37ea42294e0a049a4d10110b0ce87386a4e0f9cfffef7486ed9ca61127e8e9aa WatchSource:0}: Error finding container 37ea42294e0a049a4d10110b0ce87386a4e0f9cfffef7486ed9ca61127e8e9aa: Status 404 returned error can't find the container with id 37ea42294e0a049a4d10110b0ce87386a4e0f9cfffef7486ed9ca61127e8e9aa Jul 13 14:46:04.192851 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:04.037500 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102e87a82f37db84cece8bdb72d68ea7.slice/crio-518033919d040a4600e283d9eb6869568f30c6e31c3911060dc0b50a8a39f4f2 WatchSource:0}: Error finding container 518033919d040a4600e283d9eb6869568f30c6e31c3911060dc0b50a8a39f4f2: Status 404 returned error can't find the container with id 518033919d040a4600e283d9eb6869568f30c6e31c3911060dc0b50a8a39f4f2 Jul 13 14:46:04.192851 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.041914 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.737648 2571 apiserver.go:52] "Watching apiserver" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.746117 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.746572 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4288q","openshift-network-diagnostics/network-check-target-md84j","openshift-ovn-kubernetes/ovnkube-node-q86tn","kube-system/global-pull-secret-syncer-4dphs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc","openshift-cluster-node-tuning-operator/tuned-rm5ff","openshift-dns/node-resolver-bnfks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal","openshift-multus/multus-additional-cni-plugins-7ld4f","openshift-multus/network-metrics-daemon-lxtmc","openshift-network-operator/iptables-alerter-4vt4h","kube-system/konnectivity-agent-kxxh2","kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal","openshift-image-registry/node-ca-vnldn"] Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.749298 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.751653 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.751963 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.752124 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-brlvl\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.752134 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.754136 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rb995\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.754181 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.754275 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.755095 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.755139 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.756595 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.757123 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.757207 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.757472 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-59qsf\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.757524 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.757556 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.757655 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.758743 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.758761 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.758843 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.759648 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.762297 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.762403 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 14:46:04.774145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.762433 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-86lq2\"" Jul 13 14:46:04.775668 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.763027 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 14:46:04.775668 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.763189 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 14:46:04.775668 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.766122 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.775668 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.768095 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-cni-netd\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.775668 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.768137 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64eaf08c-2213-409c-8d19-7b470afb06ce-dbus\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.775668 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.768405 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:04.778284 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.778099 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hhkcf\"" Jul 13 14:46:04.778513 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.778361 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 14:46:04.778924 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.778577 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:46:04.778924 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.778733 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q7669\"" Jul 13 14:46:04.778924 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.778785 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 14:46:04.779361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.778960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86f3ae9-b976-415c-abbf-956a61859697-tmp-dir\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.779361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779001 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovn-node-metrics-cert\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.779361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779057 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovnkube-script-lib\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.779361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779121 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-var-lib-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.779361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-etc-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.779361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779208 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-ovn\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.779361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a86f3ae9-b976-415c-abbf-956a61859697-hosts-file\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.779854 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779529 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 14:46:04.779854 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779558 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 14:46:04.779854 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779639 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 14:46:04.779854 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779667 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 14:46:04.779854 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779732 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.779893 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6lkfj\"" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780032 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780046 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-host-slash\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbwj\" (UniqueName: \"kubernetes.io/projected/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-kube-api-access-wwbwj\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-node-log\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780305 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovnkube-config\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-systemd\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780403 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-run-ovn-kubernetes\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780443 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780480 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-env-overrides\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780514 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz44l\" (UniqueName: \"kubernetes.io/projected/66598f46-bcda-46a1-ad42-0997fe1f6a57-kube-api-access-kz44l\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780547 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-iptables-alerter-script\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-kubelet\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.933407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780605 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-log-socket\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64eaf08c-2213-409c-8d19-7b470afb06ce-kubelet-config\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780738 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6ss\" (UniqueName: \"kubernetes.io/projected/a86f3ae9-b976-415c-abbf-956a61859697-kube-api-access-xq6ss\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-systemd-units\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-slash\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.780986 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-run-netns\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.781032 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-cni-bin\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.781673 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.781754 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.784618 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.784678 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.786612 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.787039 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4288q" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.789768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.793598 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.793778 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cbl5z\"" Jul 13 14:46:04.935455 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.793830 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.793863 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ng9pb\"" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.794909 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.802695 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ztvxq" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.810223 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ztvxq" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.858676 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.868298 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" event={"ID":"102e87a82f37db84cece8bdb72d68ea7","Type":"ContainerStarted","Data":"518033919d040a4600e283d9eb6869568f30c6e31c3911060dc0b50a8a39f4f2"} Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.869525 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" event={"ID":"be6b691ccebfc90f351c1aa81ac9e62f","Type":"ContainerStarted","Data":"37ea42294e0a049a4d10110b0ce87386a4e0f9cfffef7486ed9ca61127e8e9aa"} Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-lib-modules\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-run-netns\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881848 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-kubelet\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881871 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881888 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881893 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-run-netns\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86f3ae9-b976-415c-abbf-956a61859697-tmp-dir\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovnkube-script-lib\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.936666 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881975 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-host\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.881990 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-os-release\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882010 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-device-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-etc-selinux\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882096 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-os-release\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882124 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-etc-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-ovn\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882211 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-k8s-cni-cncf-io\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882245 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a86f3ae9-b976-415c-abbf-956a61859697-hosts-file\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbwj\" (UniqueName: \"kubernetes.io/projected/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-kube-api-access-wwbwj\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882291 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-ovn\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882304 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-node-log\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882336 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovnkube-config\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882357 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a86f3ae9-b976-415c-abbf-956a61859697-hosts-file\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.937886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882363 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86f3ae9-b976-415c-abbf-956a61859697-tmp-dir\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882377 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c135566f-c963-4b21-9507-4945739c5b5a-serviceca\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882408 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-node-log\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d26ec668-e81d-41b8-9f37-48d808ce132c-agent-certs\") pod \"konnectivity-agent-kxxh2\" (UID: \"d26ec668-e81d-41b8-9f37-48d808ce132c\") " pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-env-overrides\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882487 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz44l\" (UniqueName: \"kubernetes.io/projected/66598f46-bcda-46a1-ad42-0997fe1f6a57-kube-api-access-kz44l\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882516 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txf85\" (UniqueName: \"kubernetes.io/projected/c135566f-c963-4b21-9507-4945739c5b5a-kube-api-access-txf85\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882546 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-modprobe-d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882576 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-hostroot\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882641 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cnibin\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882672 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882673 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovnkube-script-lib\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882693 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-etc-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882734 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-cnibin\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.939045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882736 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882771 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-socket-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd6hz\" (UniqueName: \"kubernetes.io/projected/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-kube-api-access-xd6hz\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-log-socket\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882882 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76e7967-ea57-4ceb-bcf8-b11226d623a0-tmp\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882892 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-env-overrides\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882901 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-cni-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882922 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-log-socket\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-cni-bin\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.882979 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovnkube-config\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.882983 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883021 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-etc-kubernetes\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.883074 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret podName:64eaf08c-2213-409c-8d19-7b470afb06ce nodeName:}" failed. No retries permitted until 2026-07-13 14:46:05.383043789 +0000 UTC m=+3.037010225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret") pod "global-pull-secret-syncer-4dphs" (UID: "64eaf08c-2213-409c-8d19-7b470afb06ce") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:04.940463 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64eaf08c-2213-409c-8d19-7b470afb06ce-kubelet-config\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883124 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysconfig\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883141 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-sys\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883168 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64eaf08c-2213-409c-8d19-7b470afb06ce-kubelet-config\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6ss\" (UniqueName: \"kubernetes.io/projected/a86f3ae9-b976-415c-abbf-956a61859697-kube-api-access-xq6ss\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-systemd-units\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883326 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-systemd-units\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-slash\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-cni-bin\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883411 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-slash\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883417 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-cni-netd\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883452 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-cni-bin\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883480 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64eaf08c-2213-409c-8d19-7b470afb06ce-dbus\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:04.941209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883534 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-cni-netd\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883548 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05b7de19-ff74-4787-a361-127c52c358b1-multus-daemon-config\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovn-node-metrics-cert\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883594 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-tuned\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883609 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pc6d\" (UniqueName: \"kubernetes.io/projected/e76e7967-ea57-4ceb-bcf8-b11226d623a0-kube-api-access-2pc6d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883656 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-socket-dir-parent\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883683 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-conf-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883708 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-multus-certs\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883717 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64eaf08c-2213-409c-8d19-7b470afb06ce-dbus\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d26ec668-e81d-41b8-9f37-48d808ce132c-konnectivity-ca\") pod \"konnectivity-agent-kxxh2\" (UID: \"d26ec668-e81d-41b8-9f37-48d808ce132c\") " pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883761 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-system-cni-dir\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-var-lib-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883816 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05b7de19-ff74-4787-a361-127c52c358b1-cni-binary-copy\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.941923 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-run\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883857 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-var-lib-openvswitch\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883867 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-var-lib-kubelet\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-host-slash\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883928 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883938 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-kubernetes\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-systemd\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-run-ovn-kubernetes\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.883987 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-host-slash\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884000 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-run-systemd\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c135566f-c963-4b21-9507-4945739c5b5a-host\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884039 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884031 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-run-ovn-kubernetes\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.942651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884052 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-cni-multus\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttfh\" (UniqueName: \"kubernetes.io/projected/05b7de19-ff74-4787-a361-127c52c358b1-kube-api-access-xttfh\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884105 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884126 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-iptables-alerter-script\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-kubelet\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysctl-conf\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884205 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66598f46-bcda-46a1-ad42-0997fe1f6a57-host-kubelet\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884215 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-systemd\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884559 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884602 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrfm\" (UniqueName: \"kubernetes.io/projected/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-kube-api-access-glrfm\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884637 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-system-cni-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-netns\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-registration-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.944079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-sys-fs\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.944799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngw98\" (UniqueName: \"kubernetes.io/projected/390e11af-ed50-4b31-9500-758b4137f35e-kube-api-access-ngw98\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:04.944799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884804 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysctl-d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:04.944799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.884921 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-iptables-alerter-script\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.944799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.890116 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66598f46-bcda-46a1-ad42-0997fe1f6a57-ovn-node-metrics-cert\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:04.944799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.899219 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbwj\" (UniqueName: \"kubernetes.io/projected/aa16c91a-ea8d-441e-bcb6-cb160f1ed146-kube-api-access-wwbwj\") pod \"iptables-alerter-4vt4h\" (UID: \"aa16c91a-ea8d-441e-bcb6-cb160f1ed146\") " pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:04.944799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.907142 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6ss\" (UniqueName: \"kubernetes.io/projected/a86f3ae9-b976-415c-abbf-956a61859697-kube-api-access-xq6ss\") pod \"node-resolver-bnfks\" (UID: \"a86f3ae9-b976-415c-abbf-956a61859697\") " pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:04.944799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.907941 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz44l\" (UniqueName: \"kubernetes.io/projected/66598f46-bcda-46a1-ad42-0997fe1f6a57-kube-api-access-kz44l\") pod \"ovnkube-node-q86tn\" (UID: \"66598f46-bcda-46a1-ad42-0997fe1f6a57\") " pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.985874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttfh\" (UniqueName: \"kubernetes.io/projected/05b7de19-ff74-4787-a361-127c52c358b1-kube-api-access-xttfh\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.985903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.985923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysctl-conf\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.985940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-systemd\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.985970 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986053 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-systemd\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrfm\" (UniqueName: \"kubernetes.io/projected/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-kube-api-access-glrfm\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986105 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysctl-conf\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986119 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-system-cni-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-netns\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-registration-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-sys-fs\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986203 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-system-cni-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.027774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986208 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-netns\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw98\" (UniqueName: \"kubernetes.io/projected/390e11af-ed50-4b31-9500-758b4137f35e-kube-api-access-ngw98\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986266 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-registration-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986270 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysctl-d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986281 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-sys-fs\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986298 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-lib-modules\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986351 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-kubelet\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986406 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986410 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-lib-modules\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986417 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysctl-d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986455 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-kubelet\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986459 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.029660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986487 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-host\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986511 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-os-release\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986554 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986579 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-host\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-device-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986594 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-os-release\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986624 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-etc-selinux\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986631 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-device-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986654 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-os-release\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986684 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-k8s-cni-cncf-io\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986680 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986714 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c135566f-c963-4b21-9507-4945739c5b5a-serviceca\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986719 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-etc-selinux\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986724 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-os-release\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.030378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986735 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d26ec668-e81d-41b8-9f37-48d808ce132c-agent-certs\") pod \"konnectivity-agent-kxxh2\" (UID: \"d26ec668-e81d-41b8-9f37-48d808ce132c\") " pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txf85\" (UniqueName: \"kubernetes.io/projected/c135566f-c963-4b21-9507-4945739c5b5a-kube-api-access-txf85\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986736 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-k8s-cni-cncf-io\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986802 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-modprobe-d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986828 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-hostroot\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986851 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cnibin\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-cnibin\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-socket-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986923 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-hostroot\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd6hz\" (UniqueName: \"kubernetes.io/projected/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-kube-api-access-xd6hz\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.986996 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76e7967-ea57-4ceb-bcf8-b11226d623a0-tmp\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987024 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-cni-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987035 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-cnibin\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-cni-bin\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987094 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-cni-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.031550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987121 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-etc-kubernetes\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987122 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/390e11af-ed50-4b31-9500-758b4137f35e-socket-dir\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987154 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c135566f-c963-4b21-9507-4945739c5b5a-serviceca\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987172 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-modprobe-d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987175 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-cni-bin\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987211 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-etc-kubernetes\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987197 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysconfig\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987184 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cnibin\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-sys\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987315 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-sysconfig\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05b7de19-ff74-4787-a361-127c52c358b1-multus-daemon-config\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987296 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-sys\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-tuned\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987378 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc6d\" (UniqueName: \"kubernetes.io/projected/e76e7967-ea57-4ceb-bcf8-b11226d623a0-kube-api-access-2pc6d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.032533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987409 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-socket-dir-parent\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987436 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-conf-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-multus-certs\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d26ec668-e81d-41b8-9f37-48d808ce132c-konnectivity-ca\") pod \"konnectivity-agent-kxxh2\" (UID: \"d26ec668-e81d-41b8-9f37-48d808ce132c\") " pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987487 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987501 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-system-cni-dir\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05b7de19-ff74-4787-a361-127c52c358b1-cni-binary-copy\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987552 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-conf-dir\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-run\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987594 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-run\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.987366 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987631 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-system-cni-dir\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987666 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-run-multus-certs\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.033255 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.987674 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:05.487654802 +0000 UTC m=+3.141621237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987874 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-multus-socket-dir-parent\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-var-lib-kubelet\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987939 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-kubernetes\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987936 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05b7de19-ff74-4787-a361-127c52c358b1-multus-daemon-config\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987972 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c135566f-c963-4b21-9507-4945739c5b5a-host\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.987998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-cni-multus\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.988027 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c135566f-c963-4b21-9507-4945739c5b5a-host\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.988029 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-kubernetes\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.988063 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05b7de19-ff74-4787-a361-127c52c358b1-host-var-lib-cni-multus\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.988068 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05b7de19-ff74-4787-a361-127c52c358b1-cni-binary-copy\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.988080 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e76e7967-ea57-4ceb-bcf8-b11226d623a0-var-lib-kubelet\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.988313 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d26ec668-e81d-41b8-9f37-48d808ce132c-konnectivity-ca\") pod \"konnectivity-agent-kxxh2\" (UID: \"d26ec668-e81d-41b8-9f37-48d808ce132c\") " pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.988378 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.989565 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d26ec668-e81d-41b8-9f37-48d808ce132c-agent-certs\") pod \"konnectivity-agent-kxxh2\" (UID: \"d26ec668-e81d-41b8-9f37-48d808ce132c\") " pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:05.034191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.989928 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76e7967-ea57-4ceb-bcf8-b11226d623a0-tmp\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.989919 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e76e7967-ea57-4ceb-bcf8-b11226d623a0-etc-tuned\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.996703 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttfh\" (UniqueName: \"kubernetes.io/projected/05b7de19-ff74-4787-a361-127c52c358b1-kube-api-access-xttfh\") pod \"multus-4288q\" (UID: \"05b7de19-ff74-4787-a361-127c52c358b1\") " pod="openshift-multus/multus-4288q" Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.997222 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.997245 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.997259 2571 projected.go:196] Error preparing data for projected volume kube-api-access-zshzq for pod openshift-network-diagnostics/network-check-target-md84j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:04.997329 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq podName:90f110a0-fcf1-4822-ade7-00dc98f9af1b nodeName:}" failed. No retries permitted until 2026-07-13 14:46:05.497312411 +0000 UTC m=+3.151278867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zshzq" (UniqueName: "kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq") pod "network-check-target-md84j" (UID: "90f110a0-fcf1-4822-ade7-00dc98f9af1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.998080 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd6hz\" (UniqueName: \"kubernetes.io/projected/acd9c4d4-669d-43f5-b072-e3aa5bd04ab8-kube-api-access-xd6hz\") pod \"multus-additional-cni-plugins-7ld4f\" (UID: \"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8\") " pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.999579 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txf85\" (UniqueName: \"kubernetes.io/projected/c135566f-c963-4b21-9507-4945739c5b5a-kube-api-access-txf85\") pod \"node-ca-vnldn\" (UID: \"c135566f-c963-4b21-9507-4945739c5b5a\") " pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:04.999795 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc6d\" (UniqueName: \"kubernetes.io/projected/e76e7967-ea57-4ceb-bcf8-b11226d623a0-kube-api-access-2pc6d\") pod \"tuned-rm5ff\" (UID: \"e76e7967-ea57-4ceb-bcf8-b11226d623a0\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.035025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.000467 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrfm\" (UniqueName: \"kubernetes.io/projected/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-kube-api-access-glrfm\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:05.117209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.062702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnfks" Jul 13 14:46:05.117209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.078789 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw98\" (UniqueName: \"kubernetes.io/projected/390e11af-ed50-4b31-9500-758b4137f35e-kube-api-access-ngw98\") pod \"aws-ebs-csi-driver-node-hd7lc\" (UID: \"390e11af-ed50-4b31-9500-758b4137f35e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.117209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.078880 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vt4h" Jul 13 14:46:05.117209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.087291 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:05.117209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.093704 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" Jul 13 14:46:05.117209 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.107087 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" Jul 13 14:46:05.119814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.117543 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnldn" Jul 13 14:46:05.128597 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.125658 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" Jul 13 14:46:05.202629 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.131666 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4288q" Jul 13 14:46:05.202629 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.137563 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:05.202629 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.145089 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:46:05.423009 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.410536 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:05.423009 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.410707 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:05.423009 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.410810 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret podName:64eaf08c-2213-409c-8d19-7b470afb06ce nodeName:}" failed. No retries permitted until 2026-07-13 14:46:06.410787987 +0000 UTC m=+4.064754427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret") pod "global-pull-secret-syncer-4dphs" (UID: "64eaf08c-2213-409c-8d19-7b470afb06ce") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.511413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.511476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.511555 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.511640 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:06.511617406 +0000 UTC m=+4.165583854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.511645 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.511664 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.511678 2571 projected.go:196] Error preparing data for projected volume kube-api-access-zshzq for pod openshift-network-diagnostics/network-check-target-md84j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:05.539953 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.511764 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq podName:90f110a0-fcf1-4822-ade7-00dc98f9af1b nodeName:}" failed. No retries permitted until 2026-07-13 14:46:06.511748781 +0000 UTC m=+4.165715217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zshzq" (UniqueName: "kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq") pod "network-check-target-md84j" (UID: "90f110a0-fcf1-4822-ade7-00dc98f9af1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:05.657190 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.546906 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b7de19_ff74_4787_a361_127c52c358b1.slice/crio-c45ea72804d6f69072be18cc7df8b588ab8fdff07c7800637221fac52f1d9c57 WatchSource:0}: Error finding container c45ea72804d6f69072be18cc7df8b588ab8fdff07c7800637221fac52f1d9c57: Status 404 returned error can't find the container with id c45ea72804d6f69072be18cc7df8b588ab8fdff07c7800637221fac52f1d9c57 Jul 13 14:46:05.657190 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.575870 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76e7967_ea57_4ceb_bcf8_b11226d623a0.slice/crio-895c2a568ea526cb1bba25a5ea5b0514159cbadcc9a1e326053c9f68a98fe4f8 WatchSource:0}: Error finding container 895c2a568ea526cb1bba25a5ea5b0514159cbadcc9a1e326053c9f68a98fe4f8: Status 404 returned error can't find the container with id 895c2a568ea526cb1bba25a5ea5b0514159cbadcc9a1e326053c9f68a98fe4f8 Jul 13 14:46:05.657190 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.576686 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd9c4d4_669d_43f5_b072_e3aa5bd04ab8.slice/crio-8390cd0133c62a1af7f302700e862ea66a2c59831b9c2ffb8988719d51f9c5a4 WatchSource:0}: Error finding container 8390cd0133c62a1af7f302700e862ea66a2c59831b9c2ffb8988719d51f9c5a4: Status 404 returned error can't find the container with id 8390cd0133c62a1af7f302700e862ea66a2c59831b9c2ffb8988719d51f9c5a4 Jul 13 14:46:05.657190 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.577748 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa16c91a_ea8d_441e_bcb6_cb160f1ed146.slice/crio-16d62032a49dce31b77425600553ebc2dc3b204877225218157e48eba69618da WatchSource:0}: Error finding container 16d62032a49dce31b77425600553ebc2dc3b204877225218157e48eba69618da: Status 404 returned error can't find the container with id 16d62032a49dce31b77425600553ebc2dc3b204877225218157e48eba69618da Jul 13 14:46:05.657190 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.579595 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390e11af_ed50_4b31_9500_758b4137f35e.slice/crio-4da0f486fea367ef38a60d2436fcd6504caa2432cee84448e0f2a59c214b8437 WatchSource:0}: Error finding container 4da0f486fea367ef38a60d2436fcd6504caa2432cee84448e0f2a59c214b8437: Status 404 returned error can't find the container with id 4da0f486fea367ef38a60d2436fcd6504caa2432cee84448e0f2a59c214b8437 Jul 13 14:46:05.813898 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.813827 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:41:05 +0000 UTC" deadline="2028-02-14 08:35:03.411231643 +0000 UTC" Jul 13 14:46:05.813898 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.813894 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13937h48m57.597341949s" Jul 13 14:46:05.864303 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.864271 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:05.864480 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:05.864412 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:05.873653 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.873620 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4288q" event={"ID":"05b7de19-ff74-4787-a361-127c52c358b1","Type":"ContainerStarted","Data":"c45ea72804d6f69072be18cc7df8b588ab8fdff07c7800637221fac52f1d9c57"} Jul 13 14:46:05.875018 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.874973 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" event={"ID":"390e11af-ed50-4b31-9500-758b4137f35e","Type":"ContainerStarted","Data":"4da0f486fea367ef38a60d2436fcd6504caa2432cee84448e0f2a59c214b8437"} Jul 13 14:46:05.876205 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.876183 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vt4h" event={"ID":"aa16c91a-ea8d-441e-bcb6-cb160f1ed146","Type":"ContainerStarted","Data":"16d62032a49dce31b77425600553ebc2dc3b204877225218157e48eba69618da"} Jul 13 14:46:05.877500 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.877476 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerStarted","Data":"8390cd0133c62a1af7f302700e862ea66a2c59831b9c2ffb8988719d51f9c5a4"} Jul 13 14:46:05.878718 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:05.878695 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" event={"ID":"e76e7967-ea57-4ceb-bcf8-b11226d623a0","Type":"ContainerStarted","Data":"895c2a568ea526cb1bba25a5ea5b0514159cbadcc9a1e326053c9f68a98fe4f8"} Jul 13 14:46:05.943226 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.943186 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86f3ae9_b976_415c_abbf_956a61859697.slice/crio-dc518513666bea6fb065a3d1833c029982b58135f4a820faeb64dc9f777ce036 WatchSource:0}: Error finding container dc518513666bea6fb065a3d1833c029982b58135f4a820faeb64dc9f777ce036: Status 404 returned error can't find the container with id dc518513666bea6fb065a3d1833c029982b58135f4a820faeb64dc9f777ce036 Jul 13 14:46:05.943947 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.943802 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26ec668_e81d_41b8_9f37_48d808ce132c.slice/crio-d5c722ca07e90a54a87db3792d975d9a97ce62363def3bd4b762e54ce8f3b25c WatchSource:0}: Error finding container d5c722ca07e90a54a87db3792d975d9a97ce62363def3bd4b762e54ce8f3b25c: Status 404 returned error can't find the container with id d5c722ca07e90a54a87db3792d975d9a97ce62363def3bd4b762e54ce8f3b25c Jul 13 14:46:05.944720 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:05.944691 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc135566f_c963_4b21_9507_4945739c5b5a.slice/crio-3cc4b1703ad2eea94d041a637f1d9cfd60c905dc6cc9b1285d9a7430c901bcc6 WatchSource:0}: Error finding container 3cc4b1703ad2eea94d041a637f1d9cfd60c905dc6cc9b1285d9a7430c901bcc6: Status 404 returned error can't find the container with id 3cc4b1703ad2eea94d041a637f1d9cfd60c905dc6cc9b1285d9a7430c901bcc6 Jul 13 14:46:06.406882 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:06.406801 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66598f46_bcda_46a1_ad42_0997fe1f6a57.slice/crio-73d6c4f668ad24c593998e9230074053db7a45adad7ac953d6ac9a621ac06029 WatchSource:0}: Error finding container 73d6c4f668ad24c593998e9230074053db7a45adad7ac953d6ac9a621ac06029: Status 404 returned error can't find the container with id 73d6c4f668ad24c593998e9230074053db7a45adad7ac953d6ac9a621ac06029 Jul 13 14:46:06.419556 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.419342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:06.419556 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.419474 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:06.419556 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.419536 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret podName:64eaf08c-2213-409c-8d19-7b470afb06ce nodeName:}" failed. No retries permitted until 2026-07-13 14:46:08.419517492 +0000 UTC m=+6.073483929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret") pod "global-pull-secret-syncer-4dphs" (UID: "64eaf08c-2213-409c-8d19-7b470afb06ce") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:06.520565 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.520525 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:06.520701 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.520637 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:06.520796 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.520776 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:06.520873 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.520852 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:08.520829909 +0000 UTC m=+6.174796342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:06.521779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.521606 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:46:06.521779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.521636 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:46:06.521779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.521650 2571 projected.go:196] Error preparing data for projected volume kube-api-access-zshzq for pod openshift-network-diagnostics/network-check-target-md84j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:06.521779 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.521708 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq podName:90f110a0-fcf1-4822-ade7-00dc98f9af1b nodeName:}" failed. No retries permitted until 2026-07-13 14:46:08.521690325 +0000 UTC m=+6.175656760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zshzq" (UniqueName: "kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq") pod "network-check-target-md84j" (UID: "90f110a0-fcf1-4822-ade7-00dc98f9af1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:06.815047 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.814924 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:41:05 +0000 UTC" deadline="2027-12-19 23:20:29.585110937 +0000 UTC" Jul 13 14:46:06.815047 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.814966 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12584h34m22.770149334s" Jul 13 14:46:06.865164 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.864371 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:06.865164 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.864511 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:06.867093 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.867065 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:06.867254 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:06.867189 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:06.897873 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.896948 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" event={"ID":"102e87a82f37db84cece8bdb72d68ea7","Type":"ContainerStarted","Data":"f1668a1d9c657976f194bd52ac0d067658a3570d67857b29702b2835c50549c8"} Jul 13 14:46:06.900727 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.900617 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"73d6c4f668ad24c593998e9230074053db7a45adad7ac953d6ac9a621ac06029"} Jul 13 14:46:06.908260 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.908136 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxxh2" event={"ID":"d26ec668-e81d-41b8-9f37-48d808ce132c","Type":"ContainerStarted","Data":"d5c722ca07e90a54a87db3792d975d9a97ce62363def3bd4b762e54ce8f3b25c"} Jul 13 14:46:06.919721 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.919683 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnldn" event={"ID":"c135566f-c963-4b21-9507-4945739c5b5a","Type":"ContainerStarted","Data":"3cc4b1703ad2eea94d041a637f1d9cfd60c905dc6cc9b1285d9a7430c901bcc6"} Jul 13 14:46:06.934790 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:06.934717 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnfks" event={"ID":"a86f3ae9-b976-415c-abbf-956a61859697","Type":"ContainerStarted","Data":"dc518513666bea6fb065a3d1833c029982b58135f4a820faeb64dc9f777ce036"} Jul 13 14:46:07.863867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:07.863826 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:07.864369 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:07.863996 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:07.961076 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:07.961022 2571 generic.go:356] "Generic (PLEG): container finished" podID="be6b691ccebfc90f351c1aa81ac9e62f" containerID="a34eac62df74672c5cb020f26cbb9ba63d4e43a2860c57784d4cc914b59de5fa" exitCode=0 Jul 13 14:46:07.961330 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:07.961301 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" event={"ID":"be6b691ccebfc90f351c1aa81ac9e62f","Type":"ContainerDied","Data":"a34eac62df74672c5cb020f26cbb9ba63d4e43a2860c57784d4cc914b59de5fa"} Jul 13 14:46:07.982571 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:07.982506 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-41.ec2.internal" podStartSLOduration=3.982482596 podStartE2EDuration="3.982482596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:06.917435807 +0000 UTC m=+4.571402266" watchObservedRunningTime="2026-07-13 14:46:07.982482596 +0000 UTC m=+5.636449045" Jul 13 14:46:08.440784 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:08.440741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:08.440965 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.440917 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:08.441046 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.440983 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret podName:64eaf08c-2213-409c-8d19-7b470afb06ce nodeName:}" failed. No retries permitted until 2026-07-13 14:46:12.440963032 +0000 UTC m=+10.094929468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret") pod "global-pull-secret-syncer-4dphs" (UID: "64eaf08c-2213-409c-8d19-7b470afb06ce") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:08.542161 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:08.542123 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:08.542351 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:08.542205 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:08.542430 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.542383 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:46:08.542430 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.542403 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:46:08.542430 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.542417 2571 projected.go:196] Error preparing data for projected volume kube-api-access-zshzq for pod openshift-network-diagnostics/network-check-target-md84j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:08.542598 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.542477 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq podName:90f110a0-fcf1-4822-ade7-00dc98f9af1b nodeName:}" failed. No retries permitted until 2026-07-13 14:46:12.542459071 +0000 UTC m=+10.196425507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zshzq" (UniqueName: "kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq") pod "network-check-target-md84j" (UID: "90f110a0-fcf1-4822-ade7-00dc98f9af1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:08.542900 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.542878 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:08.542988 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.542936 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:12.542920996 +0000 UTC m=+10.196887432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:08.865938 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:08.865327 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:08.865938 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.865457 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:08.866800 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:08.866578 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:08.866800 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:08.866687 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:09.864665 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:09.864630 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:09.864836 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:09.864787 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:10.864818 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:10.864786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:10.865457 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:10.864923 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:10.865457 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:10.864790 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:10.865457 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:10.865304 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:11.864306 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:11.864270 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:11.864483 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:11.864417 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:12.477706 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:12.477669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:12.478128 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.477843 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:12.478128 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.477918 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret podName:64eaf08c-2213-409c-8d19-7b470afb06ce nodeName:}" failed. No retries permitted until 2026-07-13 14:46:20.477894851 +0000 UTC m=+18.131861283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret") pod "global-pull-secret-syncer-4dphs" (UID: "64eaf08c-2213-409c-8d19-7b470afb06ce") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:12.578935 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:12.578896 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:12.579095 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:12.578959 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:12.579177 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.579099 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:46:12.579177 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.579120 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:46:12.579177 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.579133 2571 projected.go:196] Error preparing data for projected volume kube-api-access-zshzq for pod openshift-network-diagnostics/network-check-target-md84j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:12.579338 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.579209 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq podName:90f110a0-fcf1-4822-ade7-00dc98f9af1b nodeName:}" failed. No retries permitted until 2026-07-13 14:46:20.579189756 +0000 UTC m=+18.233156204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zshzq" (UniqueName: "kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq") pod "network-check-target-md84j" (UID: "90f110a0-fcf1-4822-ade7-00dc98f9af1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:12.579612 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.579590 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:12.579684 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.579645 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:20.579630723 +0000 UTC m=+18.233597154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:12.865901 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:12.865269 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:12.865901 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.865392 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:12.865901 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:12.865726 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:12.865901 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:12.865819 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:13.865063 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:13.864609 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:13.865063 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:13.864745 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:14.867663 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:14.867623 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:14.867663 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:14.867634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:14.868235 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:14.867748 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:14.868235 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:14.867831 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:15.864542 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:15.864508 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:15.864707 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:15.864632 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:16.866647 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:16.866618 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:16.867095 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:16.866634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:16.867224 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:16.867110 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:16.867224 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:16.867175 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:17.864693 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:17.864666 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:17.864861 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:17.864800 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:18.863821 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:18.863784 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:18.864307 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:18.863788 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:18.864307 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:18.863925 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:18.864307 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:18.863961 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:19.864369 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:19.864339 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:19.864837 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:19.864450 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:20.533467 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:20.533426 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:20.533632 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.533586 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:20.533683 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.533670 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret podName:64eaf08c-2213-409c-8d19-7b470afb06ce nodeName:}" failed. No retries permitted until 2026-07-13 14:46:36.533646007 +0000 UTC m=+34.187612465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret") pod "global-pull-secret-syncer-4dphs" (UID: "64eaf08c-2213-409c-8d19-7b470afb06ce") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:20.634595 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:20.634565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:20.634741 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:20.634608 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:20.634741 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.634721 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:46:20.634741 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.634727 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:20.634846 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.634795 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:36.634778795 +0000 UTC m=+34.288745227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:20.634846 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.634735 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:46:20.634846 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.634832 2571 projected.go:196] Error preparing data for projected volume kube-api-access-zshzq for pod openshift-network-diagnostics/network-check-target-md84j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:20.634939 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.634877 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq podName:90f110a0-fcf1-4822-ade7-00dc98f9af1b nodeName:}" failed. No retries permitted until 2026-07-13 14:46:36.634864371 +0000 UTC m=+34.288830821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zshzq" (UniqueName: "kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq") pod "network-check-target-md84j" (UID: "90f110a0-fcf1-4822-ade7-00dc98f9af1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:20.863975 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:20.863886 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:20.864137 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.864014 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:20.864137 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:20.864086 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:20.864269 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:20.864219 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:21.863878 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:21.863842 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:21.864360 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:21.863978 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:22.867025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:22.865836 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:22.867025 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:22.865988 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:22.867025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:22.866497 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:22.867025 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:22.866609 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:23.864650 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:23.864615 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:23.864805 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:23.864730 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:24.864566 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:24.864533 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:24.864886 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:24.864647 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:24.864886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:24.864713 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:24.864886 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:24.864792 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:25.864731 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:25.864404 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:25.865387 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:25.864766 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:26.006741 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.006707 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" event={"ID":"be6b691ccebfc90f351c1aa81ac9e62f","Type":"ContainerStarted","Data":"532f9bd550bb45479cee1623bfd019affaa24fe50e0282f4fd783ec318580695"} Jul 13 14:46:26.008042 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.008018 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" event={"ID":"390e11af-ed50-4b31-9500-758b4137f35e","Type":"ContainerStarted","Data":"c28864ef9df2e93098e2a21e4d62c9a43e131d21b580b31960cf0aad1907bea3"} Jul 13 14:46:26.009252 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.009228 2571 generic.go:356] "Generic (PLEG): container finished" podID="acd9c4d4-669d-43f5-b072-e3aa5bd04ab8" containerID="fe11d615339ac4a64323bc77d93d61681578cb05866bdf928962da60081ac9d5" exitCode=0 Jul 13 14:46:26.009332 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.009286 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerDied","Data":"fe11d615339ac4a64323bc77d93d61681578cb05866bdf928962da60081ac9d5"} Jul 13 14:46:26.010738 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.010717 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4288q" event={"ID":"05b7de19-ff74-4787-a361-127c52c358b1","Type":"ContainerStarted","Data":"549a796a8c0a36140335255afb88cb3363f28fd85ef546b3da8a0dfac95fb463"} Jul 13 14:46:26.013553 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.013532 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"92c7a54927e4c413eb97c9e555693857739a74d9f1ba8ec334f0560a962e8e15"} Jul 13 14:46:26.013659 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.013595 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"d7c459ac08ffc3c51bbc495e1aa92bb6eb6e6ea73a2909e8e2fd0771db03aaaf"} Jul 13 14:46:26.013659 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.013637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"1583661b449ada840ced22329bf557cf298f2b0f4d34e1f203f1891bac54f3ba"} Jul 13 14:46:26.013762 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.013674 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"47ac6f7780f2455841539cfcd03134e4025cd4eeda810f9a093c52c533843264"} Jul 13 14:46:26.013762 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.013711 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"71fabf78ec9c4111199ae8227a713800f8fbea522c75a6a26a476ac7262b9c2c"} Jul 13 14:46:26.013762 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.013759 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"2daacdf7caaf9e8ee1a917de4384ac2d7393737e3399bf58a3cfbb600120b6f6"} Jul 13 14:46:26.014874 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.014855 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxxh2" event={"ID":"d26ec668-e81d-41b8-9f37-48d808ce132c","Type":"ContainerStarted","Data":"e4cfefdfd34a8446c80517534814ab086bab79121b52ad23e61f9f05aca498a9"} Jul 13 14:46:26.016224 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.016203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" event={"ID":"e76e7967-ea57-4ceb-bcf8-b11226d623a0","Type":"ContainerStarted","Data":"e43fd0bced1e64ffa5c377128f45db135efb7fd6f53d5599cacb8da89ebcdc5b"} Jul 13 14:46:26.017445 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.017427 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnldn" event={"ID":"c135566f-c963-4b21-9507-4945739c5b5a","Type":"ContainerStarted","Data":"d19e71be8539d65a3e5a918fe018d636443f9d7fae9fc64e555f006c64f1a70f"} Jul 13 14:46:26.018596 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.018576 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnfks" event={"ID":"a86f3ae9-b976-415c-abbf-956a61859697","Type":"ContainerStarted","Data":"7b82f21815cbb90511cf0a607e5d7a9fd5a07fbc8cb85fd61acfc6399847d527"} Jul 13 14:46:26.096550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.096497 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-41.ec2.internal" podStartSLOduration=22.096481635 podStartE2EDuration="22.096481635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:26.045858955 +0000 UTC m=+23.699825408" watchObservedRunningTime="2026-07-13 14:46:26.096481635 +0000 UTC m=+23.750448088" Jul 13 14:46:26.127480 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.127396 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4288q" podStartSLOduration=3.694886322 podStartE2EDuration="23.127382981s" totalImagesPullingTime="19.432496659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:26.127317086 +0000 UTC m=+23.781283541" watchObservedRunningTime="2026-07-13 14:46:26.127382981 +0000 UTC m=+23.781349434" Jul 13 14:46:26.202731 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.202683 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bnfks" podStartSLOduration=4.159657198 podStartE2EDuration="23.202668678s" totalImagesPullingTime="19.04301148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:26.174625546 +0000 UTC m=+23.828591998" watchObservedRunningTime="2026-07-13 14:46:26.202668678 +0000 UTC m=+23.856635131" Jul 13 14:46:26.223940 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.223901 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rm5ff" podStartSLOduration=3.815036872 podStartE2EDuration="23.223887983s" totalImagesPullingTime="19.408851111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:26.202770831 +0000 UTC m=+23.856737287" watchObservedRunningTime="2026-07-13 14:46:26.223887983 +0000 UTC m=+23.877854498" Jul 13 14:46:26.224218 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.224196 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vnldn" podStartSLOduration=4.201430828 podStartE2EDuration="23.224187122s" totalImagesPullingTime="19.022756294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:26.224117755 +0000 UTC m=+23.878084208" watchObservedRunningTime="2026-07-13 14:46:26.224187122 +0000 UTC m=+23.878153577" Jul 13 14:46:26.348072 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.348044 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:26.392464 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.392405 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:26.864178 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.864097 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:26.864178 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:26.864101 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:26.864425 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:26.864216 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:26.864425 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:26.864332 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:27.022723 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:27.022693 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vt4h" event={"ID":"aa16c91a-ea8d-441e-bcb6-cb160f1ed146","Type":"ContainerStarted","Data":"fbd4b62a38f94af9b694a23a286bc4f7259650246a27381c67eec61545b7dae4"} Jul 13 14:46:27.040165 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:27.040100 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kxxh2" podStartSLOduration=4.9974470239999995 podStartE2EDuration="24.040078743s" totalImagesPullingTime="19.042631719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:26.256209176 +0000 UTC m=+23.910175629" watchObservedRunningTime="2026-07-13 14:46:27.040078743 +0000 UTC m=+24.694045201" Jul 13 14:46:27.133786 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:27.133762 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 14:46:27.864202 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:27.864168 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:27.864421 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:27.864326 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:27.896946 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:27.896834 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T14:46:27.133782621Z","UUID":"05bf5616-bec6-428c-89db-cd849d352528","Handler":null,"Name":"","Endpoint":""} Jul 13 14:46:27.898518 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:27.898491 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 14:46:27.898651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:27.898546 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 14:46:28.027449 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:28.027419 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" event={"ID":"390e11af-ed50-4b31-9500-758b4137f35e","Type":"ContainerStarted","Data":"d8874695dd545f26207b8e85e46840211e0dcc40683c0eab3b5b517360192743"} Jul 13 14:46:28.047882 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:28.047854 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:28.048491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:28.048474 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:28.073894 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:28.073697 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4vt4h" podStartSLOduration=5.686010578 podStartE2EDuration="25.073675613s" totalImagesPullingTime="19.387665035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:27.040222444 +0000 UTC m=+24.694188899" watchObservedRunningTime="2026-07-13 14:46:28.073675613 +0000 UTC m=+25.727642068" Jul 13 14:46:28.864073 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:28.864029 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:28.864377 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:28.864029 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:28.864377 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:28.864188 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:28.864377 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:28.864254 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:29.032015 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.031978 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" event={"ID":"390e11af-ed50-4b31-9500-758b4137f35e","Type":"ContainerStarted","Data":"1dfe26396afd1c0ce2bc4b313644510a1c6b16b49ec8127335becd351f36f5fa"} Jul 13 14:46:29.035049 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.035021 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"a6ab1cbd4ea4c599c5a63a4bfb94ea3ddc4f894b70cf4e3a2cd91c4106f68357"} Jul 13 14:46:29.035728 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.035699 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kxxh2" Jul 13 14:46:29.052906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.052852 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd7lc" podStartSLOduration=3.960400695 podStartE2EDuration="26.05283028s" totalImagesPullingTime="22.092429585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:29.052395327 +0000 UTC m=+26.706361780" watchObservedRunningTime="2026-07-13 14:46:29.05283028 +0000 UTC m=+26.706796735" Jul 13 14:46:29.218468 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.218377 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:29.291941 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.291900 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:29.351778 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.351750 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:29.432591 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.432544 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:29.518322 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.518231 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:29.604616 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.604584 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:29.705371 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.705332 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:29.864485 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:29.864417 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:29.864647 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:29.864537 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:30.247943 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.247895 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:30.308378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.308343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:30.668821 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.668781 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:30.769929 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.769778 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:30.864003 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.863980 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:30.864100 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.863989 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:30.864165 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:30.864118 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:30.864261 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:30.864231 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:30.875932 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.875909 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:30.977576 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:30.977544 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:31.044438 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.044194 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" event={"ID":"66598f46-bcda-46a1-ad42-0997fe1f6a57","Type":"ContainerStarted","Data":"476c9adc8df800fbc4abd1442468ee15121457523018227d368de029b9f98629"} Jul 13 14:46:31.044438 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.044282 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:31.044438 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.044330 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:31.044438 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.044373 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:31.046472 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.046446 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerStarted","Data":"e4b924d86a5f7345139523c0760647164aefab04e6ac2716cb097397b7a94b36"} Jul 13 14:46:31.059800 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.059779 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:31.059910 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.059873 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:46:31.124269 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.124178 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" podStartSLOduration=9.465925571 podStartE2EDuration="28.124163445s" totalImagesPullingTime="18.658237874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:31.085116073 +0000 UTC m=+28.739082526" watchObservedRunningTime="2026-07-13 14:46:31.124163445 +0000 UTC m=+28.778129891" Jul 13 14:46:31.465108 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.464986 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:31.551942 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.551911 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:31.853237 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.853140 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:31.864127 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.864101 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:31.864273 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:31.864223 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:31.959755 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:31.959726 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:32.050120 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:32.050092 2571 generic.go:356] "Generic (PLEG): container finished" podID="acd9c4d4-669d-43f5-b072-e3aa5bd04ab8" containerID="e4b924d86a5f7345139523c0760647164aefab04e6ac2716cb097397b7a94b36" exitCode=0 Jul 13 14:46:32.050328 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:32.050188 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerDied","Data":"e4b924d86a5f7345139523c0760647164aefab04e6ac2716cb097397b7a94b36"} Jul 13 14:46:32.866836 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:32.866811 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:32.867198 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:32.866918 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:32.867198 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:32.866967 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:32.867198 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:32.867064 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:33.028057 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:33.028026 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lxtmc"] Jul 13 14:46:33.028248 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:33.028141 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:33.028304 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:33.028259 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:33.031991 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:33.031961 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-md84j"] Jul 13 14:46:33.032563 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:33.032532 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4dphs"] Jul 13 14:46:33.052767 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:33.052740 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:33.052947 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:33.052768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:33.052947 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:33.052854 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:33.053330 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:33.053293 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:34.056727 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:34.056693 2571 generic.go:356] "Generic (PLEG): container finished" podID="acd9c4d4-669d-43f5-b072-e3aa5bd04ab8" containerID="a0301befb99013e876b78f76ceb7613b1164a3f2177682ee89c8f6e5fca8e5a9" exitCode=0 Jul 13 14:46:34.057089 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:34.056755 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerDied","Data":"a0301befb99013e876b78f76ceb7613b1164a3f2177682ee89c8f6e5fca8e5a9"} Jul 13 14:46:34.863885 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:34.863852 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:34.863885 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:34.863887 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:34.864376 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:34.863972 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:34.864376 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:34.864038 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:34.864376 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:34.864142 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:34.864376 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:34.864245 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:35.061414 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:35.061388 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerStarted","Data":"e32def85c0b78d834b5189aaa369198acf3fe8615ee1d158bc88b72bf1ea96d5"} Jul 13 14:46:36.065248 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.065215 2571 generic.go:356] "Generic (PLEG): container finished" podID="acd9c4d4-669d-43f5-b072-e3aa5bd04ab8" containerID="e32def85c0b78d834b5189aaa369198acf3fe8615ee1d158bc88b72bf1ea96d5" exitCode=0 Jul 13 14:46:36.065608 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.065273 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerDied","Data":"e32def85c0b78d834b5189aaa369198acf3fe8615ee1d158bc88b72bf1ea96d5"} Jul 13 14:46:36.543369 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.543276 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:36.543502 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.543428 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:36.543502 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.543489 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret podName:64eaf08c-2213-409c-8d19-7b470afb06ce nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.54347531 +0000 UTC m=+66.197441742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret") pod "global-pull-secret-syncer-4dphs" (UID: "64eaf08c-2213-409c-8d19-7b470afb06ce") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:46:36.643945 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.643904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:36.644145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.643964 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:36.644145 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.644078 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:36.644145 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.644147 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.644131883 +0000 UTC m=+66.298098316 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:46:36.644334 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.644088 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:46:36.644334 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.644195 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:46:36.644334 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.644209 2571 projected.go:196] Error preparing data for projected volume kube-api-access-zshzq for pod openshift-network-diagnostics/network-check-target-md84j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:36.644334 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.644265 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq podName:90f110a0-fcf1-4822-ade7-00dc98f9af1b nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.644248054 +0000 UTC m=+66.298214507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zshzq" (UniqueName: "kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq") pod "network-check-target-md84j" (UID: "90f110a0-fcf1-4822-ade7-00dc98f9af1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:46:36.868418 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.868341 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:36.868418 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.868371 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:36.868619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:36.868451 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:36.868619 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.868466 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dphs" podUID="64eaf08c-2213-409c-8d19-7b470afb06ce" Jul 13 14:46:36.868619 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.868542 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md84j" podUID="90f110a0-fcf1-4822-ade7-00dc98f9af1b" Jul 13 14:46:36.868619 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:36.868614 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxtmc" podUID="bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6" Jul 13 14:46:38.640318 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.640280 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-41.ec2.internal" event="NodeReady" Jul 13 14:46:38.640722 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.640422 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 14:46:38.684659 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.684620 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kk5pz"] Jul 13 14:46:38.688390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.688364 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b5hbv"] Jul 13 14:46:38.688551 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.688497 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.691249 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.691053 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 14:46:38.691371 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.691062 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 14:46:38.691371 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.691106 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8vvm\"" Jul 13 14:46:38.692297 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.692277 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:38.694494 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.694462 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 14:46:38.694608 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.694584 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 14:46:38.694670 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.694644 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7xw2p\"" Jul 13 14:46:38.694726 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.694716 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 14:46:38.697730 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.697707 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kk5pz"] Jul 13 14:46:38.705458 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.705435 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5hbv"] Jul 13 14:46:38.760203 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.760171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.760394 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.760268 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9027a3d-54b9-4192-a5ba-5762be52aef8-tmp-dir\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.760394 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.760288 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9027a3d-54b9-4192-a5ba-5762be52aef8-config-volume\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.760394 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.760326 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk25p\" (UniqueName: \"kubernetes.io/projected/b9027a3d-54b9-4192-a5ba-5762be52aef8-kube-api-access-dk25p\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.760394 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.760349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e-cert\") pod \"ingress-canary-b5hbv\" (UID: \"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e\") " pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:38.760608 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.760397 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvgd\" (UniqueName: \"kubernetes.io/projected/1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e-kube-api-access-lrvgd\") pod \"ingress-canary-b5hbv\" (UID: \"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e\") " pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:38.860780 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.860693 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.860780 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.860764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9027a3d-54b9-4192-a5ba-5762be52aef8-tmp-dir\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.861005 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.860791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9027a3d-54b9-4192-a5ba-5762be52aef8-config-volume\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.861005 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.860841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk25p\" (UniqueName: \"kubernetes.io/projected/b9027a3d-54b9-4192-a5ba-5762be52aef8-kube-api-access-dk25p\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.861005 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:38.860869 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:46:38.861005 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.860944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e-cert\") pod \"ingress-canary-b5hbv\" (UID: \"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e\") " pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:38.861005 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:38.860951 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls podName:b9027a3d-54b9-4192-a5ba-5762be52aef8 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:39.360929387 +0000 UTC m=+37.014895835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls") pod "dns-default-kk5pz" (UID: "b9027a3d-54b9-4192-a5ba-5762be52aef8") : secret "dns-default-metrics-tls" not found Jul 13 14:46:38.861005 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.860991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvgd\" (UniqueName: \"kubernetes.io/projected/1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e-kube-api-access-lrvgd\") pod \"ingress-canary-b5hbv\" (UID: \"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e\") " pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:38.861328 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.861179 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9027a3d-54b9-4192-a5ba-5762be52aef8-tmp-dir\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.861538 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.861509 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9027a3d-54b9-4192-a5ba-5762be52aef8-config-volume\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.864016 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.863995 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:46:38.864016 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.864015 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:46:38.864238 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.864219 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:46:38.866520 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.866500 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gzcz\"" Jul 13 14:46:38.866651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.866576 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e-cert\") pod \"ingress-canary-b5hbv\" (UID: \"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e\") " pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:38.866784 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.866736 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 14:46:38.866908 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.866806 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 14:46:38.867048 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.867030 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 14:46:38.867226 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.867086 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 14:46:38.867226 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.867201 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jvqn2\"" Jul 13 14:46:38.871877 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.871854 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvgd\" (UniqueName: \"kubernetes.io/projected/1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e-kube-api-access-lrvgd\") pod \"ingress-canary-b5hbv\" (UID: \"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e\") " pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:38.872311 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.872280 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk25p\" (UniqueName: \"kubernetes.io/projected/b9027a3d-54b9-4192-a5ba-5762be52aef8-kube-api-access-dk25p\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:38.890691 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.890656 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:38.991552 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:38.991499 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:39.009129 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:39.009093 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5hbv" Jul 13 14:46:39.365475 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:39.365426 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:39.365706 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:39.365604 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:46:39.365706 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:39.365674 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls podName:b9027a3d-54b9-4192-a5ba-5762be52aef8 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:40.36565484 +0000 UTC m=+38.019621274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls") pod "dns-default-kk5pz" (UID: "b9027a3d-54b9-4192-a5ba-5762be52aef8") : secret "dns-default-metrics-tls" not found Jul 13 14:46:40.154141 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:40.154118 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5hbv"] Jul 13 14:46:40.158785 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:46:40.158741 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c55e6c2_6bc1_4caf_b8c4_e0e654d8058e.slice/crio-0f6666e64f2e98d44bc27b532ea844c1801dfcceb462dc968a2ae4dd0b0d90e2 WatchSource:0}: Error finding container 0f6666e64f2e98d44bc27b532ea844c1801dfcceb462dc968a2ae4dd0b0d90e2: Status 404 returned error can't find the container with id 0f6666e64f2e98d44bc27b532ea844c1801dfcceb462dc968a2ae4dd0b0d90e2 Jul 13 14:46:40.373314 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:40.373281 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:40.373487 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:40.373414 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:46:40.373487 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:40.373466 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls podName:b9027a3d-54b9-4192-a5ba-5762be52aef8 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:42.373452469 +0000 UTC m=+40.027418901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls") pod "dns-default-kk5pz" (UID: "b9027a3d-54b9-4192-a5ba-5762be52aef8") : secret "dns-default-metrics-tls" not found Jul 13 14:46:41.084481 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:41.084440 2571 generic.go:356] "Generic (PLEG): container finished" podID="acd9c4d4-669d-43f5-b072-e3aa5bd04ab8" containerID="a93d8ba694c38959f8eac1f2d460b708731324827894ccff061bb2d702d118b2" exitCode=0 Jul 13 14:46:41.084672 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:41.084511 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerDied","Data":"a93d8ba694c38959f8eac1f2d460b708731324827894ccff061bb2d702d118b2"} Jul 13 14:46:41.086091 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:41.085924 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5hbv" event={"ID":"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e","Type":"ContainerStarted","Data":"0f6666e64f2e98d44bc27b532ea844c1801dfcceb462dc968a2ae4dd0b0d90e2"} Jul 13 14:46:42.092543 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:42.092504 2571 generic.go:356] "Generic (PLEG): container finished" podID="acd9c4d4-669d-43f5-b072-e3aa5bd04ab8" containerID="ab98cfc823d6e3111db1f40edee719890c83c803f904caf1883ef24563abbf59" exitCode=0 Jul 13 14:46:42.092953 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:42.092567 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerDied","Data":"ab98cfc823d6e3111db1f40edee719890c83c803f904caf1883ef24563abbf59"} Jul 13 14:46:42.389083 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:42.389002 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:42.389234 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:42.389165 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:46:42.389234 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:42.389226 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls podName:b9027a3d-54b9-4192-a5ba-5762be52aef8 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:46.389211468 +0000 UTC m=+44.043177900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls") pod "dns-default-kk5pz" (UID: "b9027a3d-54b9-4192-a5ba-5762be52aef8") : secret "dns-default-metrics-tls" not found Jul 13 14:46:43.068056 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.068021 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:43.098167 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.098110 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" event={"ID":"acd9c4d4-669d-43f5-b072-e3aa5bd04ab8","Type":"ContainerStarted","Data":"9c5b21d9c5891a4e9b1a0592fb5567150afdeb8ab7a5a2a2a1210946f48fa157"} Jul 13 14:46:43.099545 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.099520 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5hbv" event={"ID":"1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e","Type":"ContainerStarted","Data":"741abfe2eb3cf4343afed6fbf886dda47461c38cb4e55995cae6943b566d05d6"} Jul 13 14:46:43.124492 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.124452 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7ld4f" podStartSLOduration=9.871942603 podStartE2EDuration="40.12443542s" totalImagesPullingTime="30.252492817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:43.122749602 +0000 UTC m=+40.776716056" watchObservedRunningTime="2026-07-13 14:46:43.12443542 +0000 UTC m=+40.778401876" Jul 13 14:46:43.130430 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.130407 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:43.140850 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.140811 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b5hbv" podStartSLOduration=2.726167941 podStartE2EDuration="5.140797721s" totalImagesPullingTime="2.41462978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:46:43.139655142 +0000 UTC m=+40.793621595" watchObservedRunningTime="2026-07-13 14:46:43.140797721 +0000 UTC m=+40.794764211" Jul 13 14:46:43.267816 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.267784 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:43.346741 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:43.346645 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:46.418958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:46.418925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:46.419373 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:46.419070 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:46:46.419373 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:46.419142 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls podName:b9027a3d-54b9-4192-a5ba-5762be52aef8 nodeName:}" failed. No retries permitted until 2026-07-13 14:46:54.419124292 +0000 UTC m=+52.073090724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls") pod "dns-default-kk5pz" (UID: "b9027a3d-54b9-4192-a5ba-5762be52aef8") : secret "dns-default-metrics-tls" not found Jul 13 14:46:51.630219 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:51.630144 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:51.729835 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:51.729798 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:54.470247 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:46:54.470212 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:46:54.470625 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:54.470347 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:46:54.470625 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:46:54.470411 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls podName:b9027a3d-54b9-4192-a5ba-5762be52aef8 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:10.470395753 +0000 UTC m=+68.124362189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls") pod "dns-default-kk5pz" (UID: "b9027a3d-54b9-4192-a5ba-5762be52aef8") : secret "dns-default-metrics-tls" not found Jul 13 14:47:00.405641 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.405606 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-sl9b2"] Jul 13 14:47:00.407509 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.407493 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.413416 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.413377 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:47:00.414146 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.414127 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 13 14:47:00.414467 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.414448 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 13 14:47:00.415763 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.415746 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 13 14:47:00.415854 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.415775 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fk752\"" Jul 13 14:47:00.416468 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.416450 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc"] Jul 13 14:47:00.418119 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.418102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.423045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.423029 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 13 14:47:00.423415 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.423395 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 14:47:00.428204 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.428188 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 14:47:00.428372 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.428360 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g7c7z\"" Jul 13 14:47:00.437396 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.437381 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 13 14:47:00.438312 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.438293 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-sl9b2"] Jul 13 14:47:00.444404 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.444385 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc"] Jul 13 14:47:00.462009 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.461978 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 13 14:47:00.503467 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.503442 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5"] Jul 13 14:47:00.505242 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.505226 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" Jul 13 14:47:00.507696 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.507678 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5jbw2\"" Jul 13 14:47:00.508461 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.508441 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:47:00.508549 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.508477 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 13 14:47:00.511356 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.511336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c183ae1-2b41-4b01-ad48-f038e47583ed-config\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.511422 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.511370 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c183ae1-2b41-4b01-ad48-f038e47583ed-trusted-ca\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.511479 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.511419 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpwjr\" (UniqueName: \"kubernetes.io/projected/6adfc487-32f8-44ae-a5ba-a11777af7499-kube-api-access-vpwjr\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.511479 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.511445 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6adfc487-32f8-44ae-a5ba-a11777af7499-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.511479 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.511467 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpqrs\" (UniqueName: \"kubernetes.io/projected/6c183ae1-2b41-4b01-ad48-f038e47583ed-kube-api-access-wpqrs\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.511620 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.511507 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.511620 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.511534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c183ae1-2b41-4b01-ad48-f038e47583ed-serving-cert\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.518911 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.518892 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5"] Jul 13 14:47:00.608022 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.607991 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64c8c9b7db-nw27s"] Jul 13 14:47:00.609867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.609851 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.612077 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612055 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c183ae1-2b41-4b01-ad48-f038e47583ed-config\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.612264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c183ae1-2b41-4b01-ad48-f038e47583ed-trusted-ca\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.612264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpwjr\" (UniqueName: \"kubernetes.io/projected/6adfc487-32f8-44ae-a5ba-a11777af7499-kube-api-access-vpwjr\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.612264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6adfc487-32f8-44ae-a5ba-a11777af7499-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.612264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpqrs\" (UniqueName: \"kubernetes.io/projected/6c183ae1-2b41-4b01-ad48-f038e47583ed-kube-api-access-wpqrs\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.612264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.612516 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c183ae1-2b41-4b01-ad48-f038e47583ed-serving-cert\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.612516 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612281 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 13 14:47:00.612516 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612324 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24crb\" (UniqueName: \"kubernetes.io/projected/f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5-kube-api-access-24crb\") pod \"volume-data-source-validator-78495597f6-h8xp5\" (UID: \"f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" Jul 13 14:47:00.612993 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612971 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c183ae1-2b41-4b01-ad48-f038e47583ed-config\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.612993 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.612983 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6adfc487-32f8-44ae-a5ba-a11777af7499-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.613287 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:00.613267 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:00.613378 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:00.613336 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls podName:6adfc487-32f8-44ae-a5ba-a11777af7499 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:01.113317233 +0000 UTC m=+58.767283669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gdjnc" (UID: "6adfc487-32f8-44ae-a5ba-a11777af7499") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:00.613653 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.613624 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w86r5\"" Jul 13 14:47:00.613745 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.613706 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 13 14:47:00.613806 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.613787 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c183ae1-2b41-4b01-ad48-f038e47583ed-trusted-ca\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.613874 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.613813 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 13 14:47:00.613874 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.613843 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 14:47:00.613874 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.613851 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 13 14:47:00.613874 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.613862 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 14:47:00.617036 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.617015 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c183ae1-2b41-4b01-ad48-f038e47583ed-serving-cert\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.625752 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.625732 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64c8c9b7db-nw27s"] Jul 13 14:47:00.629504 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.629484 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpqrs\" (UniqueName: \"kubernetes.io/projected/6c183ae1-2b41-4b01-ad48-f038e47583ed-kube-api-access-wpqrs\") pod \"console-operator-7866497494-sl9b2\" (UID: \"6c183ae1-2b41-4b01-ad48-f038e47583ed\") " pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.629705 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.629684 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpwjr\" (UniqueName: \"kubernetes.io/projected/6adfc487-32f8-44ae-a5ba-a11777af7499-kube-api-access-vpwjr\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:00.712997 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.712913 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.712997 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.712953 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-stats-auth\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.712997 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.712996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-default-certificate\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.713236 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.713020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdp5\" (UniqueName: \"kubernetes.io/projected/c9cb46c8-1226-48ff-9cb5-27df8b576c32-kube-api-access-zgdp5\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.713236 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.713039 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24crb\" (UniqueName: \"kubernetes.io/projected/f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5-kube-api-access-24crb\") pod \"volume-data-source-validator-78495597f6-h8xp5\" (UID: \"f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" Jul 13 14:47:00.713236 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.713109 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.715962 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.715944 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:00.724019 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.723992 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24crb\" (UniqueName: \"kubernetes.io/projected/f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5-kube-api-access-24crb\") pod \"volume-data-source-validator-78495597f6-h8xp5\" (UID: \"f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" Jul 13 14:47:00.813850 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.813822 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" Jul 13 14:47:00.814010 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.813992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.814073 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.814037 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.814073 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.814062 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-stats-auth\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.814198 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.814096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-default-certificate\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.814198 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.814128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdp5\" (UniqueName: \"kubernetes.io/projected/c9cb46c8-1226-48ff-9cb5-27df8b576c32-kube-api-access-zgdp5\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.814292 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:00.814225 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:47:00.814292 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:00.814229 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:01.314209223 +0000 UTC m=+58.968175656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : configmap references non-existent config key: service-ca.crt Jul 13 14:47:00.814375 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:00.814303 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:01.314284806 +0000 UTC m=+58.968251243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : secret "router-metrics-certs-default" not found Jul 13 14:47:00.816512 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.816495 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-default-certificate\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.816643 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.816621 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-stats-auth\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.825270 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.825249 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdp5\" (UniqueName: \"kubernetes.io/projected/c9cb46c8-1226-48ff-9cb5-27df8b576c32-kube-api-access-zgdp5\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:00.845914 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.845883 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-sl9b2"] Jul 13 14:47:00.848393 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:00.848360 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c183ae1_2b41_4b01_ad48_f038e47583ed.slice/crio-7e1411ad2f65d689f79df551052ed4c20363653d00c3b5be1d965ea20a8dba29 WatchSource:0}: Error finding container 7e1411ad2f65d689f79df551052ed4c20363653d00c3b5be1d965ea20a8dba29: Status 404 returned error can't find the container with id 7e1411ad2f65d689f79df551052ed4c20363653d00c3b5be1d965ea20a8dba29 Jul 13 14:47:00.945812 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:00.945778 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5"] Jul 13 14:47:00.949731 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:00.949701 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cdb5b8_7b4d_4dd1_8588_5c0cb8b2bcc5.slice/crio-fd7e1341aaca951709409a259c74ee4d959ae401364804e59dad622fabf53f02 WatchSource:0}: Error finding container fd7e1341aaca951709409a259c74ee4d959ae401364804e59dad622fabf53f02: Status 404 returned error can't find the container with id fd7e1341aaca951709409a259c74ee4d959ae401364804e59dad622fabf53f02 Jul 13 14:47:01.116133 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:01.116102 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:01.116329 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:01.116271 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:01.116368 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:01.116344 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls podName:6adfc487-32f8-44ae-a5ba-a11777af7499 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:02.116325859 +0000 UTC m=+59.770292291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gdjnc" (UID: "6adfc487-32f8-44ae-a5ba-a11777af7499") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:01.141305 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:01.141273 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-sl9b2" event={"ID":"6c183ae1-2b41-4b01-ad48-f038e47583ed","Type":"ContainerStarted","Data":"7e1411ad2f65d689f79df551052ed4c20363653d00c3b5be1d965ea20a8dba29"} Jul 13 14:47:01.142328 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:01.142306 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" event={"ID":"f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5","Type":"ContainerStarted","Data":"fd7e1341aaca951709409a259c74ee4d959ae401364804e59dad622fabf53f02"} Jul 13 14:47:01.318024 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:01.317987 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:01.318264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:01.318037 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:01.318264 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:01.318186 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:02.318165509 +0000 UTC m=+59.972131953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : configmap references non-existent config key: service-ca.crt Jul 13 14:47:01.318264 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:01.318187 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:47:01.318264 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:01.318252 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:02.318231609 +0000 UTC m=+59.972198056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : secret "router-metrics-certs-default" not found Jul 13 14:47:02.123286 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:02.123252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:02.123735 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:02.123389 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:02.123735 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:02.123448 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls podName:6adfc487-32f8-44ae-a5ba-a11777af7499 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:04.123434104 +0000 UTC m=+61.777400541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gdjnc" (UID: "6adfc487-32f8-44ae-a5ba-a11777af7499") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:02.325375 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:02.325338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:02.325592 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:02.325406 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:02.325592 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:02.325493 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:47:02.325592 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:02.325547 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:04.325533287 +0000 UTC m=+61.979499735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : configmap references non-existent config key: service-ca.crt Jul 13 14:47:02.325592 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:02.325562 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:04.325555873 +0000 UTC m=+61.979522304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : secret "router-metrics-certs-default" not found Jul 13 14:47:03.070846 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:03.070815 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q86tn" Jul 13 14:47:03.149943 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:03.149908 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" event={"ID":"f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5","Type":"ContainerStarted","Data":"984c0b530282280e158947667661c10d5613f1a72c21f129b3270eb50a3d7b65"} Jul 13 14:47:04.139792 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:04.139747 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:04.139982 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:04.139938 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:04.140049 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:04.140036 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls podName:6adfc487-32f8-44ae-a5ba-a11777af7499 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.140012745 +0000 UTC m=+65.793979206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gdjnc" (UID: "6adfc487-32f8-44ae-a5ba-a11777af7499") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:04.341066 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:04.341020 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:04.341550 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:04.341188 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:04.341550 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:04.341240 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:47:04.341550 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:04.341325 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.341302477 +0000 UTC m=+65.995268923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : secret "router-metrics-certs-default" not found Jul 13 14:47:04.341550 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:04.341360 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.341340937 +0000 UTC m=+65.995307382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : configmap references non-existent config key: service-ca.crt Jul 13 14:47:05.156377 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.156349 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/0.log" Jul 13 14:47:05.156575 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.156389 2571 generic.go:356] "Generic (PLEG): container finished" podID="6c183ae1-2b41-4b01-ad48-f038e47583ed" containerID="0bd5ee5c65deef55529ac4125e032781b69d4219388a467b7cd172878a9a0fc1" exitCode=255 Jul 13 14:47:05.156575 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.156431 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-sl9b2" event={"ID":"6c183ae1-2b41-4b01-ad48-f038e47583ed","Type":"ContainerDied","Data":"0bd5ee5c65deef55529ac4125e032781b69d4219388a467b7cd172878a9a0fc1"} Jul 13 14:47:05.156740 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.156713 2571 scope.go:117] "RemoveContainer" containerID="0bd5ee5c65deef55529ac4125e032781b69d4219388a467b7cd172878a9a0fc1" Jul 13 14:47:05.172828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.172780 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-h8xp5" podStartSLOduration=3.780302645 podStartE2EDuration="5.172760461s" totalImagesPullingTime="1.392457816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:03.169977128 +0000 UTC m=+60.823943581" watchObservedRunningTime="2026-07-13 14:47:05.172760461 +0000 UTC m=+62.826726917" Jul 13 14:47:05.173259 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.173241 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-crrw5"] Jul 13 14:47:05.177164 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.177127 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.180360 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.180341 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 14:47:05.180645 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.180623 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 14:47:05.180645 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.180637 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 14:47:05.180775 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.180652 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dhzjh\"" Jul 13 14:47:05.180775 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.180648 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 14:47:05.185944 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.185924 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-crrw5"] Jul 13 14:47:05.247878 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.247839 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.248028 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.247933 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1b26389-1c68-47e6-9201-ab14423132a7-data-volume\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.248028 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.247996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1b26389-1c68-47e6-9201-ab14423132a7-crio-socket\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.248131 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.248062 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgr8\" (UniqueName: \"kubernetes.io/projected/f1b26389-1c68-47e6-9201-ab14423132a7-kube-api-access-5dgr8\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.248131 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.248117 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.349451 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.349413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.349472 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1b26389-1c68-47e6-9201-ab14423132a7-data-volume\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.349497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1b26389-1c68-47e6-9201-ab14423132a7-crio-socket\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.349520 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgr8\" (UniqueName: \"kubernetes.io/projected/f1b26389-1c68-47e6-9201-ab14423132a7-kube-api-access-5dgr8\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.349559 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.349613 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.349615 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1b26389-1c68-47e6-9201-ab14423132a7-crio-socket\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.349633 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:05.849612258 +0000 UTC m=+63.503578708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : secret "insights-runtime-extractor-tls" not found Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.349568 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.349819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.349663 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:05.84964787 +0000 UTC m=+63.503614316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : configmap "kube-rbac-proxy" not found Jul 13 14:47:05.350123 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.349886 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1b26389-1c68-47e6-9201-ab14423132a7-data-volume\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.359704 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.359686 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgr8\" (UniqueName: \"kubernetes.io/projected/f1b26389-1c68-47e6-9201-ab14423132a7-kube-api-access-5dgr8\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.853308 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.853273 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.853485 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:05.853333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:05.853485 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.853389 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:47:05.853485 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.853450 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:06.853434697 +0000 UTC m=+64.507401129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : configmap "kube-rbac-proxy" not found Jul 13 14:47:05.853485 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.853453 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:47:05.853640 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:05.853496 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:06.853481178 +0000 UTC m=+64.507447611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : secret "insights-runtime-extractor-tls" not found Jul 13 14:47:06.160424 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.160347 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 14:47:06.160818 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.160801 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/0.log" Jul 13 14:47:06.160879 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.160836 2571 generic.go:356] "Generic (PLEG): container finished" podID="6c183ae1-2b41-4b01-ad48-f038e47583ed" containerID="0924226d57a5de6282dbedd74f99f27298252bafaef80084890f6f55ddf1a6e9" exitCode=255 Jul 13 14:47:06.160879 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.160862 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-sl9b2" event={"ID":"6c183ae1-2b41-4b01-ad48-f038e47583ed","Type":"ContainerDied","Data":"0924226d57a5de6282dbedd74f99f27298252bafaef80084890f6f55ddf1a6e9"} Jul 13 14:47:06.160946 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.160914 2571 scope.go:117] "RemoveContainer" containerID="0bd5ee5c65deef55529ac4125e032781b69d4219388a467b7cd172878a9a0fc1" Jul 13 14:47:06.161136 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.161120 2571 scope.go:117] "RemoveContainer" containerID="0924226d57a5de6282dbedd74f99f27298252bafaef80084890f6f55ddf1a6e9" Jul 13 14:47:06.161328 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:06.161305 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-sl9b2_openshift-console-operator(6c183ae1-2b41-4b01-ad48-f038e47583ed)\"" pod="openshift-console-operator/console-operator-7866497494-sl9b2" podUID="6c183ae1-2b41-4b01-ad48-f038e47583ed" Jul 13 14:47:06.861694 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.861659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:06.862128 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:06.861801 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:47:06.862128 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:06.861879 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.861864215 +0000 UTC m=+66.515830647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : secret "insights-runtime-extractor-tls" not found Jul 13 14:47:06.862128 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.861872 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:06.862128 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:06.861987 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:47:06.862128 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:06.862045 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:08.862030547 +0000 UTC m=+66.515996980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : configmap "kube-rbac-proxy" not found Jul 13 14:47:06.957785 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:06.957755 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnfks_a86f3ae9-b976-415c-abbf-956a61859697/dns-node-resolver/0.log" Jul 13 14:47:07.164869 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:07.164795 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 14:47:07.165097 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:07.165083 2571 scope.go:117] "RemoveContainer" containerID="0924226d57a5de6282dbedd74f99f27298252bafaef80084890f6f55ddf1a6e9" Jul 13 14:47:07.165253 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:07.165235 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-sl9b2_openshift-console-operator(6c183ae1-2b41-4b01-ad48-f038e47583ed)\"" pod="openshift-console-operator/console-operator-7866497494-sl9b2" podUID="6c183ae1-2b41-4b01-ad48-f038e47583ed" Jul 13 14:47:08.175559 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.175523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:08.175962 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.175662 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:08.175962 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.175730 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls podName:6adfc487-32f8-44ae-a5ba-a11777af7499 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:16.175715134 +0000 UTC m=+73.829681567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gdjnc" (UID: "6adfc487-32f8-44ae-a5ba-a11777af7499") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:08.353781 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.353754 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnldn_c135566f-c963-4b21-9507-4945739c5b5a/node-ca/0.log" Jul 13 14:47:08.377176 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.377134 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:08.377286 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.377203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:08.377345 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.377302 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:47:08.377345 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.377334 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:16.377319566 +0000 UTC m=+74.031285998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : configmap references non-existent config key: service-ca.crt Jul 13 14:47:08.377455 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.377362 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:16.377342883 +0000 UTC m=+74.031309324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : secret "router-metrics-certs-default" not found Jul 13 14:47:08.578937 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.578904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:47:08.581929 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.581908 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 14:47:08.591776 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.591748 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64eaf08c-2213-409c-8d19-7b470afb06ce-original-pull-secret\") pod \"global-pull-secret-syncer-4dphs\" (UID: \"64eaf08c-2213-409c-8d19-7b470afb06ce\") " pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:47:08.679432 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.679393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:47:08.679561 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.679465 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:47:08.682544 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.682523 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 14:47:08.682658 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.682569 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 14:47:08.689955 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.689932 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 14:47:08.690052 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.690002 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs podName:bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6 nodeName:}" failed. No retries permitted until 2026-07-13 14:48:12.689981027 +0000 UTC m=+130.343947459 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs") pod "network-metrics-daemon-lxtmc" (UID: "bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6") : secret "metrics-daemon-secret" not found Jul 13 14:47:08.692779 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.692760 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 14:47:08.702405 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.702376 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshzq\" (UniqueName: \"kubernetes.io/projected/90f110a0-fcf1-4822-ade7-00dc98f9af1b-kube-api-access-zshzq\") pod \"network-check-target-md84j\" (UID: \"90f110a0-fcf1-4822-ade7-00dc98f9af1b\") " pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:47:08.881228 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.881128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:08.881386 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.881260 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:08.881386 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.881278 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:47:08.881386 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.881344 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:12.881329592 +0000 UTC m=+70.535296025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : secret "insights-runtime-extractor-tls" not found Jul 13 14:47:08.881386 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.881380 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:47:08.881559 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:08.881434 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:12.881421467 +0000 UTC m=+70.535387899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : configmap "kube-rbac-proxy" not found Jul 13 14:47:08.888032 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.888008 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dphs" Jul 13 14:47:08.906043 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.906020 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jvqn2\"" Jul 13 14:47:08.913868 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.913848 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:47:08.959583 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:08.959556 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5hbv_1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e/serve-healthcheck-canary/0.log" Jul 13 14:47:09.040415 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:09.040384 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4dphs"] Jul 13 14:47:09.043696 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:09.043673 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64eaf08c_2213_409c_8d19_7b470afb06ce.slice/crio-d166c69648382d51ebecfb1e99ca2569e90bc41cad4baaa3143266b99a682eaa WatchSource:0}: Error finding container d166c69648382d51ebecfb1e99ca2569e90bc41cad4baaa3143266b99a682eaa: Status 404 returned error can't find the container with id d166c69648382d51ebecfb1e99ca2569e90bc41cad4baaa3143266b99a682eaa Jul 13 14:47:09.057466 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:09.057443 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-md84j"] Jul 13 14:47:09.062302 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:09.062276 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f110a0_fcf1_4822_ade7_00dc98f9af1b.slice/crio-dc1f3a8115bf6aa3244502f8eda0c452d6af5dcfdfc66fe67f9124b0b2c5e44c WatchSource:0}: Error finding container dc1f3a8115bf6aa3244502f8eda0c452d6af5dcfdfc66fe67f9124b0b2c5e44c: Status 404 returned error can't find the container with id dc1f3a8115bf6aa3244502f8eda0c452d6af5dcfdfc66fe67f9124b0b2c5e44c Jul 13 14:47:09.171267 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:09.171175 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-md84j" event={"ID":"90f110a0-fcf1-4822-ade7-00dc98f9af1b","Type":"ContainerStarted","Data":"dc1f3a8115bf6aa3244502f8eda0c452d6af5dcfdfc66fe67f9124b0b2c5e44c"} Jul 13 14:47:09.172189 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:09.172143 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4dphs" event={"ID":"64eaf08c-2213-409c-8d19-7b470afb06ce","Type":"ContainerStarted","Data":"d166c69648382d51ebecfb1e99ca2569e90bc41cad4baaa3143266b99a682eaa"} Jul 13 14:47:10.494762 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:10.494722 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:47:10.495258 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:10.494893 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:47:10.495258 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:10.494973 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls podName:b9027a3d-54b9-4192-a5ba-5762be52aef8 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:42.494954241 +0000 UTC m=+100.148920689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls") pod "dns-default-kk5pz" (UID: "b9027a3d-54b9-4192-a5ba-5762be52aef8") : secret "dns-default-metrics-tls" not found Jul 13 14:47:10.716695 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:10.716648 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:10.716882 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:10.716721 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:10.717143 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:10.717120 2571 scope.go:117] "RemoveContainer" containerID="0924226d57a5de6282dbedd74f99f27298252bafaef80084890f6f55ddf1a6e9" Jul 13 14:47:10.717358 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:10.717324 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-sl9b2_openshift-console-operator(6c183ae1-2b41-4b01-ad48-f038e47583ed)\"" pod="openshift-console-operator/console-operator-7866497494-sl9b2" podUID="6c183ae1-2b41-4b01-ad48-f038e47583ed" Jul 13 14:47:12.915326 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:12.915291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:12.915819 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:12.915374 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:12.915819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:12.915432 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:47:12.915819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:12.915506 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:20.915482837 +0000 UTC m=+78.569449291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : configmap "kube-rbac-proxy" not found Jul 13 14:47:12.915819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:12.915516 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:47:12.915819 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:12.915570 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:20.915554714 +0000 UTC m=+78.569521148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : secret "insights-runtime-extractor-tls" not found Jul 13 14:47:14.187643 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:14.187606 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-md84j" event={"ID":"90f110a0-fcf1-4822-ade7-00dc98f9af1b","Type":"ContainerStarted","Data":"3ae83f8c211e677abcdcf8c627fb498c4f2b62bc6c7eadf0640fc8b6bef7f04f"} Jul 13 14:47:14.188122 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:14.187716 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:47:14.189273 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:14.189248 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4dphs" event={"ID":"64eaf08c-2213-409c-8d19-7b470afb06ce","Type":"ContainerStarted","Data":"ed6a88c42718e222aa245ca6f2c214c12a03172f6669e6b0aaa48b3c92cd875d"} Jul 13 14:47:14.206015 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:14.205969 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-md84j" podStartSLOduration=66.529229948 podStartE2EDuration="1m11.205956029s" totalImagesPullingTime="4.676726081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:14.205462286 +0000 UTC m=+71.859428737" watchObservedRunningTime="2026-07-13 14:47:14.205956029 +0000 UTC m=+71.859922482" Jul 13 14:47:14.222430 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:14.222381 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4dphs" podStartSLOduration=66.523052865 podStartE2EDuration="1m11.222368245s" totalImagesPullingTime="4.69931538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:14.221449163 +0000 UTC m=+71.875415617" watchObservedRunningTime="2026-07-13 14:47:14.222368245 +0000 UTC m=+71.876334698" Jul 13 14:47:16.080689 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.080655 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-rw27m"] Jul 13 14:47:16.083620 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.083602 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.086226 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.086197 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 13 14:47:16.086332 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.086225 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 13 14:47:16.086332 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.086233 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4529p\"" Jul 13 14:47:16.087208 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.087187 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 13 14:47:16.087342 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.087324 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 13 14:47:16.092474 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.092454 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-rw27m"] Jul 13 14:47:16.140554 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.140522 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-signing-cabundle\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.140685 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.140567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvvx\" (UniqueName: \"kubernetes.io/projected/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-kube-api-access-clvvx\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.140685 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.140592 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-signing-key\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.240971 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.240936 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-signing-cabundle\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.241126 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.240986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvvx\" (UniqueName: \"kubernetes.io/projected/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-kube-api-access-clvvx\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.241126 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.241010 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-signing-key\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.241126 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.241064 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:16.241271 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:16.241194 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:16.241271 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:16.241256 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls podName:6adfc487-32f8-44ae-a5ba-a11777af7499 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:32.241239923 +0000 UTC m=+89.895206355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gdjnc" (UID: "6adfc487-32f8-44ae-a5ba-a11777af7499") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:47:16.241698 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.241674 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-signing-cabundle\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.243431 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.243410 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-signing-key\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.250577 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.250553 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvvx\" (UniqueName: \"kubernetes.io/projected/99e68c48-42a5-42b1-b7b8-0e8042c9a5eb-kube-api-access-clvvx\") pod \"service-ca-79f68b9f4f-rw27m\" (UID: \"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.392638 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.392555 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" Jul 13 14:47:16.443276 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.443242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:16.443417 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.443330 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:16.443478 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:16.443441 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:32.4434176 +0000 UTC m=+90.097384049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : configmap references non-existent config key: service-ca.crt Jul 13 14:47:16.443478 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:16.443468 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:47:16.443586 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:16.443522 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs podName:c9cb46c8-1226-48ff-9cb5-27df8b576c32 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:32.443505013 +0000 UTC m=+90.097471446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs") pod "router-default-64c8c9b7db-nw27s" (UID: "c9cb46c8-1226-48ff-9cb5-27df8b576c32") : secret "router-metrics-certs-default" not found Jul 13 14:47:16.512689 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:16.512657 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-rw27m"] Jul 13 14:47:16.515587 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:16.515561 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e68c48_42a5_42b1_b7b8_0e8042c9a5eb.slice/crio-8a00f8e8fd7fbc766890f8c94425e80cc241934451768c7f70ddc3ff0801ef45 WatchSource:0}: Error finding container 8a00f8e8fd7fbc766890f8c94425e80cc241934451768c7f70ddc3ff0801ef45: Status 404 returned error can't find the container with id 8a00f8e8fd7fbc766890f8c94425e80cc241934451768c7f70ddc3ff0801ef45 Jul 13 14:47:17.197369 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:17.197333 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" event={"ID":"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb","Type":"ContainerStarted","Data":"8a00f8e8fd7fbc766890f8c94425e80cc241934451768c7f70ddc3ff0801ef45"} Jul 13 14:47:19.205304 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:19.205264 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" event={"ID":"99e68c48-42a5-42b1-b7b8-0e8042c9a5eb","Type":"ContainerStarted","Data":"67c0cff3d49c9ac14377c14bd9f456e8810deccefabe0b513097690294ba116c"} Jul 13 14:47:19.223506 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:19.223453 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-rw27m" podStartSLOduration=1.429218316 podStartE2EDuration="3.223437847s" totalImagesPullingTime="1.794219531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:19.223224656 +0000 UTC m=+76.877191111" watchObservedRunningTime="2026-07-13 14:47:19.223437847 +0000 UTC m=+76.877404331" Jul 13 14:47:20.980937 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:20.980898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:20.981410 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:20.980962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:20.981410 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:20.981083 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:47:20.981410 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:20.981180 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls podName:f1b26389-1c68-47e6-9201-ab14423132a7 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:36.981136427 +0000 UTC m=+94.635102862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crrw5" (UID: "f1b26389-1c68-47e6-9201-ab14423132a7") : secret "insights-runtime-extractor-tls" not found Jul 13 14:47:20.981520 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:20.981479 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1b26389-1c68-47e6-9201-ab14423132a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:24.864279 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:24.864241 2571 scope.go:117] "RemoveContainer" containerID="0924226d57a5de6282dbedd74f99f27298252bafaef80084890f6f55ddf1a6e9" Jul 13 14:47:25.225065 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:25.224939 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 14:47:25.225065 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:25.225017 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-sl9b2" event={"ID":"6c183ae1-2b41-4b01-ad48-f038e47583ed","Type":"ContainerStarted","Data":"97a4f00855c76cf42aaa5e0667e71a1444b6ac7bf6af7e66e48c1c20f804aaf3"} Jul 13 14:47:25.225305 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:25.225284 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:25.245196 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:25.245131 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-sl9b2" podStartSLOduration=21.485325332 podStartE2EDuration="25.2451182s" totalImagesPullingTime="3.759792868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:25.243591789 +0000 UTC m=+82.897558256" watchObservedRunningTime="2026-07-13 14:47:25.2451182 +0000 UTC m=+82.899084653" Jul 13 14:47:25.859049 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:25.859019 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-sl9b2" Jul 13 14:47:32.272685 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.272646 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:32.275028 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.275005 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6adfc487-32f8-44ae-a5ba-a11777af7499-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gdjnc\" (UID: \"6adfc487-32f8-44ae-a5ba-a11777af7499\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:32.474904 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.474864 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:32.475071 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.474923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:32.475609 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.475589 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cb46c8-1226-48ff-9cb5-27df8b576c32-service-ca-bundle\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:32.477345 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.477329 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cb46c8-1226-48ff-9cb5-27df8b576c32-metrics-certs\") pod \"router-default-64c8c9b7db-nw27s\" (UID: \"c9cb46c8-1226-48ff-9cb5-27df8b576c32\") " pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:32.528590 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.528514 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g7c7z\"" Jul 13 14:47:32.536530 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.536509 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" Jul 13 14:47:32.660881 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.660856 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc"] Jul 13 14:47:32.663247 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:32.663216 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adfc487_32f8_44ae_a5ba_a11777af7499.slice/crio-5c3ee35f2d0b26e22553093c24618d1a191f0fc6da5d25e23fb0d60966de3ccd WatchSource:0}: Error finding container 5c3ee35f2d0b26e22553093c24618d1a191f0fc6da5d25e23fb0d60966de3ccd: Status 404 returned error can't find the container with id 5c3ee35f2d0b26e22553093c24618d1a191f0fc6da5d25e23fb0d60966de3ccd Jul 13 14:47:32.726326 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.726300 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w86r5\"" Jul 13 14:47:32.734651 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.734628 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:32.861294 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:32.861251 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64c8c9b7db-nw27s"] Jul 13 14:47:32.865539 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:32.865509 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cb46c8_1226_48ff_9cb5_27df8b576c32.slice/crio-75caa601ac63febee3fd0b723400ed7c15db25602136fc7eba021651444a8fb4 WatchSource:0}: Error finding container 75caa601ac63febee3fd0b723400ed7c15db25602136fc7eba021651444a8fb4: Status 404 returned error can't find the container with id 75caa601ac63febee3fd0b723400ed7c15db25602136fc7eba021651444a8fb4 Jul 13 14:47:33.250329 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:33.250293 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" event={"ID":"6adfc487-32f8-44ae-a5ba-a11777af7499","Type":"ContainerStarted","Data":"5c3ee35f2d0b26e22553093c24618d1a191f0fc6da5d25e23fb0d60966de3ccd"} Jul 13 14:47:33.251599 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:33.251569 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" event={"ID":"c9cb46c8-1226-48ff-9cb5-27df8b576c32","Type":"ContainerStarted","Data":"acd351e39262dfd2800b77208649612628273c9fcb60334b6855e2ff7db76ac9"} Jul 13 14:47:33.251757 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:33.251610 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" event={"ID":"c9cb46c8-1226-48ff-9cb5-27df8b576c32","Type":"ContainerStarted","Data":"75caa601ac63febee3fd0b723400ed7c15db25602136fc7eba021651444a8fb4"} Jul 13 14:47:33.272479 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:33.272434 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" podStartSLOduration=33.272420979 podStartE2EDuration="33.272420979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:33.27185333 +0000 UTC m=+90.925819784" watchObservedRunningTime="2026-07-13 14:47:33.272420979 +0000 UTC m=+90.926387432" Jul 13 14:47:33.735124 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:33.735077 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:33.737599 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:33.737575 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:34.255496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:34.255465 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:34.256721 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:34.256701 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64c8c9b7db-nw27s" Jul 13 14:47:35.260640 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:35.260554 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" event={"ID":"6adfc487-32f8-44ae-a5ba-a11777af7499","Type":"ContainerStarted","Data":"1d2cac3a3271abe0e38e3e4a4bb9fe4e6212bbc88b94fcd81b4c0c7578d836ee"} Jul 13 14:47:35.282136 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:35.282088 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gdjnc" podStartSLOduration=33.049889201 podStartE2EDuration="35.282073256s" totalImagesPullingTime="2.232184055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:35.279553851 +0000 UTC m=+92.933520306" watchObservedRunningTime="2026-07-13 14:47:35.282073256 +0000 UTC m=+92.936039709" Jul 13 14:47:37.009362 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:37.009321 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:37.011555 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:37.011535 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1b26389-1c68-47e6-9201-ab14423132a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crrw5\" (UID: \"f1b26389-1c68-47e6-9201-ab14423132a7\") " pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:37.311435 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:37.311394 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-crrw5" Jul 13 14:47:37.439876 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:37.439844 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-crrw5"] Jul 13 14:47:37.443059 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:37.443032 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b26389_1c68_47e6_9201_ab14423132a7.slice/crio-aa95c48b1c884236e0e55f60906aa3470369d67ab9e9ba2cf1999245179a88c1 WatchSource:0}: Error finding container aa95c48b1c884236e0e55f60906aa3470369d67ab9e9ba2cf1999245179a88c1: Status 404 returned error can't find the container with id aa95c48b1c884236e0e55f60906aa3470369d67ab9e9ba2cf1999245179a88c1 Jul 13 14:47:38.271711 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.271669 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crrw5" event={"ID":"f1b26389-1c68-47e6-9201-ab14423132a7","Type":"ContainerStarted","Data":"aa95c48b1c884236e0e55f60906aa3470369d67ab9e9ba2cf1999245179a88c1"} Jul 13 14:47:38.365085 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.364985 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq"] Jul 13 14:47:38.369284 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.368529 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.372601 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.371498 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 14:47:38.373531 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.373110 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qjvfq\"" Jul 13 14:47:38.373531 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.373300 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 14:47:38.373531 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.373378 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 13 14:47:38.374304 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.373563 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 14:47:38.384777 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.384724 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq"] Jul 13 14:47:38.480717 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.480683 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6558cc6799-2pq6d"] Jul 13 14:47:38.483999 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.483970 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.489618 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.489593 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 14:47:38.489752 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.489630 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 14:47:38.490415 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.490389 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 14:47:38.490547 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.490529 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 14:47:38.490659 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.490643 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 14:47:38.490767 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.490752 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 14:47:38.490866 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.490830 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5hvf\"" Jul 13 14:47:38.490945 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.490930 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 14:47:38.496930 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.496904 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79d4bdbdbc-cs5pk"] Jul 13 14:47:38.500431 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.500347 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6558cc6799-2pq6d"] Jul 13 14:47:38.500543 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.500523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.503755 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.503001 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 14:47:38.503755 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.503346 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7ljvr\"" Jul 13 14:47:38.503755 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.503369 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 14:47:38.507995 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.507972 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 14:47:38.510330 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.510311 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 14:47:38.516374 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.516341 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d4bdbdbc-cs5pk"] Jul 13 14:47:38.521271 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.521239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73e4f249-e7f2-4c96-87c2-d3ae671bc49d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66475f5f5d-5bwjq\" (UID: \"73e4f249-e7f2-4c96-87c2-d3ae671bc49d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.521427 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.521304 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7744\" (UniqueName: \"kubernetes.io/projected/73e4f249-e7f2-4c96-87c2-d3ae671bc49d-kube-api-access-k7744\") pod \"managed-serviceaccount-addon-agent-66475f5f5d-5bwjq\" (UID: \"73e4f249-e7f2-4c96-87c2-d3ae671bc49d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.621978 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.621943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc633b5-112a-4736-989d-863fa5480d0f-registry-certificates\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.621996 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7744\" (UniqueName: \"kubernetes.io/projected/73e4f249-e7f2-4c96-87c2-d3ae671bc49d-kube-api-access-k7744\") pod \"managed-serviceaccount-addon-agent-66475f5f5d-5bwjq\" (UID: \"73e4f249-e7f2-4c96-87c2-d3ae671bc49d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622036 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc633b5-112a-4736-989d-863fa5480d0f-installation-pull-secrets\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622065 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-config\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622116 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc633b5-112a-4736-989d-863fa5480d0f-image-registry-private-configuration\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622140 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-bound-sa-token\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622182 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc633b5-112a-4736-989d-863fa5480d0f-ca-trust-extracted\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622280 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc633b5-112a-4736-989d-863fa5480d0f-trusted-ca\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622321 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-serving-cert\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-oauth-config\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622395 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-service-ca\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622424 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmznt\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-kube-api-access-xmznt\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.622491 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622470 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-registry-tls\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.623138 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622503 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-oauth-serving-cert\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.623138 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622573 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73e4f249-e7f2-4c96-87c2-d3ae671bc49d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66475f5f5d-5bwjq\" (UID: \"73e4f249-e7f2-4c96-87c2-d3ae671bc49d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.623138 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.622623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwvr\" (UniqueName: \"kubernetes.io/projected/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-kube-api-access-rvwvr\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.626914 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.626867 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73e4f249-e7f2-4c96-87c2-d3ae671bc49d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66475f5f5d-5bwjq\" (UID: \"73e4f249-e7f2-4c96-87c2-d3ae671bc49d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.634236 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.634212 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7744\" (UniqueName: \"kubernetes.io/projected/73e4f249-e7f2-4c96-87c2-d3ae671bc49d-kube-api-access-k7744\") pod \"managed-serviceaccount-addon-agent-66475f5f5d-5bwjq\" (UID: \"73e4f249-e7f2-4c96-87c2-d3ae671bc49d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.689438 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.689404 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" Jul 13 14:47:38.723036 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723000 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc633b5-112a-4736-989d-863fa5480d0f-ca-trust-extracted\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723233 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc633b5-112a-4736-989d-863fa5480d0f-trusted-ca\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723233 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723085 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-serving-cert\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.723233 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723127 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-oauth-config\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.723233 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-service-ca\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.723233 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmznt\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-kube-api-access-xmznt\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723245 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-registry-tls\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723270 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-oauth-serving-cert\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwvr\" (UniqueName: \"kubernetes.io/projected/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-kube-api-access-rvwvr\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc633b5-112a-4736-989d-863fa5480d0f-registry-certificates\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc633b5-112a-4736-989d-863fa5480d0f-installation-pull-secrets\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723404 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-config\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723443 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc633b5-112a-4736-989d-863fa5480d0f-ca-trust-extracted\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723496 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc633b5-112a-4736-989d-863fa5480d0f-image-registry-private-configuration\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.723928 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.723508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-bound-sa-token\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.724585 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.724557 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-oauth-serving-cert\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.724778 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.724753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc633b5-112a-4736-989d-863fa5480d0f-registry-certificates\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.725197 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.725075 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc633b5-112a-4736-989d-863fa5480d0f-trusted-ca\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.725785 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.725407 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-service-ca\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.725785 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.725749 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-config\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.726432 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.726386 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc633b5-112a-4736-989d-863fa5480d0f-image-registry-private-configuration\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.727738 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.727697 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-registry-tls\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.727831 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.727756 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-serving-cert\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.727934 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.727899 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc633b5-112a-4736-989d-863fa5480d0f-installation-pull-secrets\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.728485 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.728460 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-oauth-config\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.736462 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.736419 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwvr\" (UniqueName: \"kubernetes.io/projected/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-kube-api-access-rvwvr\") pod \"console-6558cc6799-2pq6d\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.736566 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.736502 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-bound-sa-token\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.737764 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.737741 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmznt\" (UniqueName: \"kubernetes.io/projected/bbc633b5-112a-4736-989d-863fa5480d0f-kube-api-access-xmznt\") pod \"image-registry-79d4bdbdbc-cs5pk\" (UID: \"bbc633b5-112a-4736-989d-863fa5480d0f\") " pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.797536 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.797299 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:38.815579 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.815537 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:38.845317 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.845280 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq"] Jul 13 14:47:38.849213 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:38.849181 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73e4f249_e7f2_4c96_87c2_d3ae671bc49d.slice/crio-5d983adbf8d9604490d9363a17bf6939c130e66c1076063a1402511452444404 WatchSource:0}: Error finding container 5d983adbf8d9604490d9363a17bf6939c130e66c1076063a1402511452444404: Status 404 returned error can't find the container with id 5d983adbf8d9604490d9363a17bf6939c130e66c1076063a1402511452444404 Jul 13 14:47:38.958355 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.958295 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6558cc6799-2pq6d"] Jul 13 14:47:38.963237 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:38.963208 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7228ecf9_0de3_4cb4_ab32_8e36de4fbce3.slice/crio-5d866c97ab0731966eb868b7f2d9f1fd9f42e300ff2e3812258f6d3697994812 WatchSource:0}: Error finding container 5d866c97ab0731966eb868b7f2d9f1fd9f42e300ff2e3812258f6d3697994812: Status 404 returned error can't find the container with id 5d866c97ab0731966eb868b7f2d9f1fd9f42e300ff2e3812258f6d3697994812 Jul 13 14:47:38.982969 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:38.982939 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d4bdbdbc-cs5pk"] Jul 13 14:47:38.986052 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:38.986023 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc633b5_112a_4736_989d_863fa5480d0f.slice/crio-a537968da76f4cb469861fb09837ef4986b2c6c03a98d94f5f29675df0cc10ee WatchSource:0}: Error finding container a537968da76f4cb469861fb09837ef4986b2c6c03a98d94f5f29675df0cc10ee: Status 404 returned error can't find the container with id a537968da76f4cb469861fb09837ef4986b2c6c03a98d94f5f29675df0cc10ee Jul 13 14:47:39.277574 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:39.277493 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" event={"ID":"73e4f249-e7f2-4c96-87c2-d3ae671bc49d","Type":"ContainerStarted","Data":"5d983adbf8d9604490d9363a17bf6939c130e66c1076063a1402511452444404"} Jul 13 14:47:39.279377 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:39.279345 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" event={"ID":"bbc633b5-112a-4736-989d-863fa5480d0f","Type":"ContainerStarted","Data":"1a123dea931338da82fc8d2120f14e69c20e1db1dd661b1263fe3e3d0425190f"} Jul 13 14:47:39.279590 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:39.279400 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" event={"ID":"bbc633b5-112a-4736-989d-863fa5480d0f","Type":"ContainerStarted","Data":"a537968da76f4cb469861fb09837ef4986b2c6c03a98d94f5f29675df0cc10ee"} Jul 13 14:47:39.279590 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:39.279463 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:47:39.280967 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:39.280946 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6558cc6799-2pq6d" event={"ID":"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3","Type":"ContainerStarted","Data":"5d866c97ab0731966eb868b7f2d9f1fd9f42e300ff2e3812258f6d3697994812"} Jul 13 14:47:39.299685 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:39.299629 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" podStartSLOduration=1.299607132 podStartE2EDuration="1.299607132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:39.298932986 +0000 UTC m=+96.952899440" watchObservedRunningTime="2026-07-13 14:47:39.299607132 +0000 UTC m=+96.953573587" Jul 13 14:47:42.559016 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:42.558981 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:47:42.561164 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:42.561134 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9027a3d-54b9-4192-a5ba-5762be52aef8-metrics-tls\") pod \"dns-default-kk5pz\" (UID: \"b9027a3d-54b9-4192-a5ba-5762be52aef8\") " pod="openshift-dns/dns-default-kk5pz" Jul 13 14:47:42.604512 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:42.604483 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8vvm\"" Jul 13 14:47:42.612659 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:42.612634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kk5pz" Jul 13 14:47:44.130638 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:44.130610 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kk5pz"] Jul 13 14:47:44.134332 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:44.134282 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9027a3d_54b9_4192_a5ba_5762be52aef8.slice/crio-0ffb6f9e32fa100efc78369183e186bbd6a457d5636ee13fd825bf7e73736175 WatchSource:0}: Error finding container 0ffb6f9e32fa100efc78369183e186bbd6a457d5636ee13fd825bf7e73736175: Status 404 returned error can't find the container with id 0ffb6f9e32fa100efc78369183e186bbd6a457d5636ee13fd825bf7e73736175 Jul 13 14:47:44.301333 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:44.301297 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" event={"ID":"73e4f249-e7f2-4c96-87c2-d3ae671bc49d","Type":"ContainerStarted","Data":"ddbde68532bc6e0bbe7d48ed84ba2b75286a4fb098b9ca2f60b817cd34a20cff"} Jul 13 14:47:44.302439 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:44.302416 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kk5pz" event={"ID":"b9027a3d-54b9-4192-a5ba-5762be52aef8","Type":"ContainerStarted","Data":"0ffb6f9e32fa100efc78369183e186bbd6a457d5636ee13fd825bf7e73736175"} Jul 13 14:47:44.303756 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:44.303732 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crrw5" event={"ID":"f1b26389-1c68-47e6-9201-ab14423132a7","Type":"ContainerStarted","Data":"24d287acd993b702c4b5940fd43b5b633a0c12f6af605fa78bf705ed0c72f813"} Jul 13 14:47:44.305140 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:44.305114 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6558cc6799-2pq6d" event={"ID":"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3","Type":"ContainerStarted","Data":"d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00"} Jul 13 14:47:44.318393 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:44.318100 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66475f5f5d-5bwjq" podStartSLOduration=1.176611408 podStartE2EDuration="6.318082786s" totalImagesPullingTime="5.141471378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:44.317426229 +0000 UTC m=+101.971392682" watchObservedRunningTime="2026-07-13 14:47:44.318082786 +0000 UTC m=+101.972049241" Jul 13 14:47:44.335910 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:44.335874 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6558cc6799-2pq6d" podStartSLOduration=1.25230391 podStartE2EDuration="6.335861369s" totalImagesPullingTime="5.083557459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:44.334996249 +0000 UTC m=+101.988962704" watchObservedRunningTime="2026-07-13 14:47:44.335861369 +0000 UTC m=+101.989827822" Jul 13 14:47:45.194559 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:45.194528 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-md84j" Jul 13 14:47:45.310006 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:45.309964 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crrw5" event={"ID":"f1b26389-1c68-47e6-9201-ab14423132a7","Type":"ContainerStarted","Data":"2319647138bb4cb9c5ba1d2b5bac2740e1ca875bdfe035005badbff80af50e85"} Jul 13 14:47:47.318101 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:47.318059 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kk5pz" event={"ID":"b9027a3d-54b9-4192-a5ba-5762be52aef8","Type":"ContainerStarted","Data":"934a87578fe988f94cb947036ae4d7770bd52972b8ee2f9bd114052475dd79c2"} Jul 13 14:47:47.318623 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:47.318125 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kk5pz" event={"ID":"b9027a3d-54b9-4192-a5ba-5762be52aef8","Type":"ContainerStarted","Data":"faf4f070962daf7327f6e1c2cdb8aef4141b9ed1e622e9381a00297e948cfea7"} Jul 13 14:47:47.318623 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:47.318511 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kk5pz" Jul 13 14:47:47.339785 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:47.339720 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kk5pz" podStartSLOduration=67.259894593 podStartE2EDuration="1m9.339701341s" totalImagesPullingTime="2.079806748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:47.33790164 +0000 UTC m=+104.991868104" watchObservedRunningTime="2026-07-13 14:47:47.339701341 +0000 UTC m=+104.993667798" Jul 13 14:47:48.798010 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.797964 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:48.798449 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.798040 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:48.803003 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.802979 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:48.922462 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.922431 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5"] Jul 13 14:47:48.938655 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.938630 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:48.941177 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.941117 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5"] Jul 13 14:47:48.942458 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.942283 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 13 14:47:48.942606 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.942586 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 13 14:47:48.943060 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.943025 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fx6vh\"" Jul 13 14:47:48.943459 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.943443 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 14:47:48.945856 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.944529 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-mk6zw"] Jul 13 14:47:48.964967 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.964939 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:48.969292 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.968416 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-mk6zw"] Jul 13 14:47:48.969292 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.968623 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wfgc9\"" Jul 13 14:47:48.969292 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.968727 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 13 14:47:48.969292 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.968827 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 13 14:47:48.969292 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.969040 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 13 14:47:48.983585 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:48.983547 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s22nb"] Jul 13 14:47:49.001243 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.001214 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.005392 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.005368 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 14:47:49.005842 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.005824 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 14:47:49.006076 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.006058 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qt79p\"" Jul 13 14:47:49.006415 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.006397 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 14:47:49.117289 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117261 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.117457 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117318 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35d475a6-d857-4d42-917b-c0e797398243-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.117457 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117345 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-sys\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117457 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117396 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45bd\" (UniqueName: \"kubernetes.io/projected/35d475a6-d857-4d42-917b-c0e797398243-kube-api-access-s45bd\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.117619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117487 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdjb\" (UniqueName: \"kubernetes.io/projected/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-api-access-7mdjb\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.117619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.117619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-wtmp\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117554 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.117619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-tls\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e09e1d-0940-4bd7-b785-74e40ee40e14-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-metrics-client-ca\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9w5j\" (UniqueName: \"kubernetes.io/projected/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-kube-api-access-r9w5j\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117745 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-textfile\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e5e09e1d-0940-4bd7-b785-74e40ee40e14-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117794 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-accelerators-collector-config\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.117958 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.117814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-root\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.218913 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.218879 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.218913 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.218925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-wtmp\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219176 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219056 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.219176 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219070 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-wtmp\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219176 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219107 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-tls\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219176 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.219396 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219180 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e09e1d-0940-4bd7-b785-74e40ee40e14-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.219396 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-metrics-client-ca\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219396 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.219396 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219307 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9w5j\" (UniqueName: \"kubernetes.io/projected/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-kube-api-access-r9w5j\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219396 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219336 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-textfile\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219396 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e5e09e1d-0940-4bd7-b785-74e40ee40e14-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-accelerators-collector-config\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219451 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-root\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219490 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219545 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35d475a6-d857-4d42-917b-c0e797398243-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219568 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-sys\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219592 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45bd\" (UniqueName: \"kubernetes.io/projected/35d475a6-d857-4d42-917b-c0e797398243-kube-api-access-s45bd\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219629 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.219677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdjb\" (UniqueName: \"kubernetes.io/projected/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-api-access-7mdjb\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.220030 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.219910 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-metrics-client-ca\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.220030 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:49.220009 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 14:47:49.220133 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:49.220078 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-tls podName:4c9bb98e-e2c7-4eb5-8468-2c8fda939998 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:49.720056994 +0000 UTC m=+107.374023433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-tls") pod "node-exporter-s22nb" (UID: "4c9bb98e-e2c7-4eb5-8468-2c8fda939998") : secret "node-exporter-tls" not found Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.220299 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-root\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:49.220364 2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:47:49.220410 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-tls podName:35d475a6-d857-4d42-917b-c0e797398243 nodeName:}" failed. No retries permitted until 2026-07-13 14:47:49.720395404 +0000 UTC m=+107.374361855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-2lfc5" (UID: "35d475a6-d857-4d42-917b-c0e797398243") : secret "openshift-state-metrics-tls" not found Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.220739 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.220782 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-textfile\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.220962 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e09e1d-0940-4bd7-b785-74e40ee40e14-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.221031 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-sys\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.221073 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e5e09e1d-0940-4bd7-b785-74e40ee40e14-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.221615 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.221573 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-accelerators-collector-config\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.222191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.221670 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35d475a6-d857-4d42-917b-c0e797398243-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.225015 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.224971 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.225308 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.225259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.225454 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.225427 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.225543 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.225471 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.230347 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.230313 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdjb\" (UniqueName: \"kubernetes.io/projected/e5e09e1d-0940-4bd7-b785-74e40ee40e14-kube-api-access-7mdjb\") pod \"kube-state-metrics-785778ff74-mk6zw\" (UID: \"e5e09e1d-0940-4bd7-b785-74e40ee40e14\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.230964 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.230926 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9w5j\" (UniqueName: \"kubernetes.io/projected/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-kube-api-access-r9w5j\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.235510 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.235484 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45bd\" (UniqueName: \"kubernetes.io/projected/35d475a6-d857-4d42-917b-c0e797398243-kube-api-access-s45bd\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.285071 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.285038 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" Jul 13 14:47:49.328528 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.328320 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crrw5" event={"ID":"f1b26389-1c68-47e6-9201-ab14423132a7","Type":"ContainerStarted","Data":"392abba321b4792d15e962dd2f42bebc84761893655cc334301355e6a1c7cf47"} Jul 13 14:47:49.333886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.333825 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:47:49.360404 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.360346 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-crrw5" podStartSLOduration=32.940790003000004 podStartE2EDuration="44.360329024s" totalImagesPullingTime="11.419539021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:47:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:49.359029112 +0000 UTC m=+107.012995560" watchObservedRunningTime="2026-07-13 14:47:49.360329024 +0000 UTC m=+107.014295480" Jul 13 14:47:49.471114 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.471083 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-mk6zw"] Jul 13 14:47:49.475619 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:49.475590 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e09e1d_0940_4bd7_b785_74e40ee40e14.slice/crio-56a8f764727a9bcfe5026927fc94bd289e10aa608f66e7756596ee3a1a8c5072 WatchSource:0}: Error finding container 56a8f764727a9bcfe5026927fc94bd289e10aa608f66e7756596ee3a1a8c5072: Status 404 returned error can't find the container with id 56a8f764727a9bcfe5026927fc94bd289e10aa608f66e7756596ee3a1a8c5072 Jul 13 14:47:49.724842 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.724746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-tls\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.724842 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.724792 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.727098 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.727066 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c9bb98e-e2c7-4eb5-8468-2c8fda939998-node-exporter-tls\") pod \"node-exporter-s22nb\" (UID: \"4c9bb98e-e2c7-4eb5-8468-2c8fda939998\") " pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.727304 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.727284 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35d475a6-d857-4d42-917b-c0e797398243-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-2lfc5\" (UID: \"35d475a6-d857-4d42-917b-c0e797398243\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.853116 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.853077 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" Jul 13 14:47:49.914025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.913918 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s22nb" Jul 13 14:47:49.927394 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:49.927364 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9bb98e_e2c7_4eb5_8468_2c8fda939998.slice/crio-44208e7e6fa2705b206d9ca996d2e60be40fcb693ec0bc99e3c984260c1262f5 WatchSource:0}: Error finding container 44208e7e6fa2705b206d9ca996d2e60be40fcb693ec0bc99e3c984260c1262f5: Status 404 returned error can't find the container with id 44208e7e6fa2705b206d9ca996d2e60be40fcb693ec0bc99e3c984260c1262f5 Jul 13 14:47:49.995053 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:49.994976 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5"] Jul 13 14:47:49.998224 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:49.998196 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d475a6_d857_4d42_917b_c0e797398243.slice/crio-b07414e8cf4c1eea7a7b11094615d14ec13ef3f6760aa1aa416ba68d71fde7d0 WatchSource:0}: Error finding container b07414e8cf4c1eea7a7b11094615d14ec13ef3f6760aa1aa416ba68d71fde7d0: Status 404 returned error can't find the container with id b07414e8cf4c1eea7a7b11094615d14ec13ef3f6760aa1aa416ba68d71fde7d0 Jul 13 14:47:50.058731 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.058707 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 14:47:50.062678 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.062649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.070459 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.070432 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 13 14:47:50.074706 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.074429 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 13 14:47:50.074706 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.074465 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 13 14:47:50.074706 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.074518 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 13 14:47:50.076193 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.075884 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 13 14:47:50.076409 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.076345 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 13 14:47:50.076589 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.076552 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 13 14:47:50.076774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.076638 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 13 14:47:50.076774 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.076761 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k7rz6\"" Jul 13 14:47:50.077041 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.077026 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 13 14:47:50.086189 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.086079 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 14:47:50.227696 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.227660 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.227888 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.227704 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.227888 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.227831 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.227888 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.227879 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228064 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.227902 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228064 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228005 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228064 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228235 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228078 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228235 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228102 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228235 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228167 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pj8j\" (UniqueName: \"kubernetes.io/projected/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-kube-api-access-5pj8j\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228235 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228219 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228405 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228251 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.228405 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.228283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.328808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.328808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.328808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328803 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.329100 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.329100 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328865 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.329100 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328918 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.329100 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328943 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.329100 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.328974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pj8j\" (UniqueName: \"kubernetes.io/projected/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-kube-api-access-5pj8j\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330013 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330081 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330094 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330179 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330217 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330466 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.330936 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.330568 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.332182 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.332135 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.332735 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.332684 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.333891 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.333868 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.335933 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.335692 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.335933 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.335902 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.336541 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.336499 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.336642 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.336623 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.337062 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.337030 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.337178 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.337066 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.339419 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.339354 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s22nb" event={"ID":"4c9bb98e-e2c7-4eb5-8468-2c8fda939998","Type":"ContainerStarted","Data":"44208e7e6fa2705b206d9ca996d2e60be40fcb693ec0bc99e3c984260c1262f5"} Jul 13 14:47:50.342196 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.342124 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" event={"ID":"35d475a6-d857-4d42-917b-c0e797398243","Type":"ContainerStarted","Data":"2a345460d03f8c02ee0ca3cb9ea639f1ef1f89e53280d418141cc9704c8fb83e"} Jul 13 14:47:50.342293 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.342200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" event={"ID":"35d475a6-d857-4d42-917b-c0e797398243","Type":"ContainerStarted","Data":"262a997d71974f48944ffcfb83e539e65ca926a543a6a537a5a140d9009a659f"} Jul 13 14:47:50.342293 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.342229 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" event={"ID":"35d475a6-d857-4d42-917b-c0e797398243","Type":"ContainerStarted","Data":"b07414e8cf4c1eea7a7b11094615d14ec13ef3f6760aa1aa416ba68d71fde7d0"} Jul 13 14:47:50.344362 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.344290 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" event={"ID":"e5e09e1d-0940-4bd7-b785-74e40ee40e14","Type":"ContainerStarted","Data":"56a8f764727a9bcfe5026927fc94bd289e10aa608f66e7756596ee3a1a8c5072"} Jul 13 14:47:50.347639 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.347611 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pj8j\" (UniqueName: \"kubernetes.io/projected/0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb-kube-api-access-5pj8j\") pod \"alertmanager-main-0\" (UID: \"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.406379 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.406348 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 14:47:50.583886 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:50.583857 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 14:47:50.586886 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:47:50.586851 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4f3e95_0a3d_4f3a_80bf_ad1af2c29ccb.slice/crio-1fd56b9b2d86f01ed4395d26d00a502652e5456f13d7f00edca68a53f7e11af5 WatchSource:0}: Error finding container 1fd56b9b2d86f01ed4395d26d00a502652e5456f13d7f00edca68a53f7e11af5: Status 404 returned error can't find the container with id 1fd56b9b2d86f01ed4395d26d00a502652e5456f13d7f00edca68a53f7e11af5 Jul 13 14:47:51.350321 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:51.350271 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerStarted","Data":"1fd56b9b2d86f01ed4395d26d00a502652e5456f13d7f00edca68a53f7e11af5"} Jul 13 14:47:52.355035 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.354932 2571 generic.go:356] "Generic (PLEG): container finished" podID="4c9bb98e-e2c7-4eb5-8468-2c8fda939998" containerID="eb2221994054e42b5af30adf87a4163953ee09b8aa8a87aefd8b5eb7a53f11e9" exitCode=0 Jul 13 14:47:52.355035 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.355020 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s22nb" event={"ID":"4c9bb98e-e2c7-4eb5-8468-2c8fda939998","Type":"ContainerDied","Data":"eb2221994054e42b5af30adf87a4163953ee09b8aa8a87aefd8b5eb7a53f11e9"} Jul 13 14:47:52.357559 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.357526 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" event={"ID":"35d475a6-d857-4d42-917b-c0e797398243","Type":"ContainerStarted","Data":"a40bca5113aa47c191acdd1acac834a8dda34bb2d1058de9cfe4dd2750aaa66b"} Jul 13 14:47:52.359922 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.359893 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" event={"ID":"e5e09e1d-0940-4bd7-b785-74e40ee40e14","Type":"ContainerStarted","Data":"dc30f1d4fa8debe51920be13ef8ca82e915eac459964e9e9af9e0f4fdc4e41a5"} Jul 13 14:47:52.360038 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.359952 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" event={"ID":"e5e09e1d-0940-4bd7-b785-74e40ee40e14","Type":"ContainerStarted","Data":"48d324483744c818b8a43f962cad03010f0161e03c036e2bd00c89812fb38642"} Jul 13 14:47:52.360038 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.359975 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" event={"ID":"e5e09e1d-0940-4bd7-b785-74e40ee40e14","Type":"ContainerStarted","Data":"ab82837aa9275fc0bb8c59e6b39579b484e868f9e93e4ce13427098b13c68157"} Jul 13 14:47:52.361531 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.361504 2571 generic.go:356] "Generic (PLEG): container finished" podID="0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb" containerID="b9330a782d54795513b23a5e128ec6292e152bda39b66a0bd969598b67061bd7" exitCode=0 Jul 13 14:47:52.361640 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.361583 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerDied","Data":"b9330a782d54795513b23a5e128ec6292e152bda39b66a0bd969598b67061bd7"} Jul 13 14:47:52.506916 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.506856 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-mk6zw" podStartSLOduration=2.429970258 podStartE2EDuration="4.506838283s" totalImagesPullingTime="2.076868025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:52.505471996 +0000 UTC m=+110.159438463" watchObservedRunningTime="2026-07-13 14:47:52.506838283 +0000 UTC m=+110.160804736" Jul 13 14:47:52.531865 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:52.531812 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-2lfc5" podStartSLOduration=3.117022188 podStartE2EDuration="4.531796317s" totalImagesPullingTime="1.414774129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:52.530463869 +0000 UTC m=+110.184430322" watchObservedRunningTime="2026-07-13 14:47:52.531796317 +0000 UTC m=+110.185762771" Jul 13 14:47:53.367689 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:53.367653 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s22nb" event={"ID":"4c9bb98e-e2c7-4eb5-8468-2c8fda939998","Type":"ContainerStarted","Data":"410f009cf6a1dd5d78b954d76cba6ef10966ceb4737907651614b58b86b3d34f"} Jul 13 14:47:53.368092 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:53.367744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s22nb" event={"ID":"4c9bb98e-e2c7-4eb5-8468-2c8fda939998","Type":"ContainerStarted","Data":"ea6d6df18b6dc79b396e8c4100a0bb098d68337bd6993cad37851650980a3cd1"} Jul 13 14:47:53.390140 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:53.390079 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s22nb" podStartSLOduration=3.63186895 podStartE2EDuration="5.390063487s" totalImagesPullingTime="1.625241103s" totalInitContainerRuntime="132.953434ms" isStatefulPod=true podCreationTimestamp="2026-07-13 14:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:53.388645861 +0000 UTC m=+111.042612313" watchObservedRunningTime="2026-07-13 14:47:53.390063487 +0000 UTC m=+111.044029940" Jul 13 14:47:53.845445 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:53.845399 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6558cc6799-2pq6d"] Jul 13 14:47:54.375133 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:54.375102 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerStarted","Data":"7612037e49f018ff3b29840b579cefe2628c6553529204f9693d60efe36c950f"} Jul 13 14:47:54.375481 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:54.375186 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerStarted","Data":"0b33c3ea7e4d6fb69d4993a11b7abd5b8cd09e2c5b14621714da4460d2635ba8"} Jul 13 14:47:54.375481 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:54.375213 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerStarted","Data":"3fce1952da71bc5750cf86887c70dff282114ea915a93fe22c856d9a3d053d86"} Jul 13 14:47:54.375481 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:54.375237 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerStarted","Data":"34e23e16e9ed0c9891bd962995fefb7965c7eef904cbbcbda31dd99ece306fa8"} Jul 13 14:47:54.375481 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:54.375261 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerStarted","Data":"d517defa7f5cfd7a45760643f2c783b8140283ba3e9454b36fc7de4ecbfa6758"} Jul 13 14:47:56.386816 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:56.386773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb","Type":"ContainerStarted","Data":"8d007c872d7b4df61d0a6f3dd7b5c5d96a44f79289d7c8fb68f5c8794f1919fe"} Jul 13 14:47:56.421314 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:56.421263 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.357425448 podStartE2EDuration="6.421248097s" totalImagesPullingTime="4.025150221s" totalInitContainerRuntime="38.672428ms" isStatefulPod=false podCreationTimestamp="2026-07-13 14:47:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:47:56.420385653 +0000 UTC m=+114.074352108" watchObservedRunningTime="2026-07-13 14:47:56.421248097 +0000 UTC m=+114.075214551" Jul 13 14:47:57.323703 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:47:57.323677 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kk5pz" Jul 13 14:48:00.289192 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:00.289139 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79d4bdbdbc-cs5pk" Jul 13 14:48:12.726753 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:12.726707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:48:12.729014 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:12.728992 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6-metrics-certs\") pod \"network-metrics-daemon-lxtmc\" (UID: \"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6\") " pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:48:12.800124 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:12.800092 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gzcz\"" Jul 13 14:48:12.807519 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:12.807500 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxtmc" Jul 13 14:48:13.105482 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:13.105444 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lxtmc"] Jul 13 14:48:13.109416 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:48:13.109389 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf85a4f_0bcd_4da3_9fec_17d0e07c81e6.slice/crio-a08b67117be2136517c6c4d251cc9ebba70a6bb44d6855eeff9ca63b47f9d60a WatchSource:0}: Error finding container a08b67117be2136517c6c4d251cc9ebba70a6bb44d6855eeff9ca63b47f9d60a: Status 404 returned error can't find the container with id a08b67117be2136517c6c4d251cc9ebba70a6bb44d6855eeff9ca63b47f9d60a Jul 13 14:48:13.448142 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:13.448106 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lxtmc" event={"ID":"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6","Type":"ContainerStarted","Data":"a08b67117be2136517c6c4d251cc9ebba70a6bb44d6855eeff9ca63b47f9d60a"} Jul 13 14:48:15.461587 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:15.461541 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lxtmc" event={"ID":"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6","Type":"ContainerStarted","Data":"28b8a0eb880ffc34cee81cdcaba69951deba6d2708d0f12a8f225249fcb82a52"} Jul 13 14:48:15.462001 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:15.461635 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lxtmc" event={"ID":"bdf85a4f-0bcd-4da3-9fec-17d0e07c81e6","Type":"ContainerStarted","Data":"a79a4aaede6207287709b8efedac817e4d77c70572ff24d09e32bf059da2e6a4"} Jul 13 14:48:15.489336 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:15.489284 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lxtmc" podStartSLOduration=131.103662414 podStartE2EDuration="2m12.48926689s" totalImagesPullingTime="1.385604476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:48:15.488076114 +0000 UTC m=+133.142042569" watchObservedRunningTime="2026-07-13 14:48:15.48926689 +0000 UTC m=+133.143233343" Jul 13 14:48:15.876706 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:15.876680 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8c9b7db-nw27s_c9cb46c8-1226-48ff-9cb5-27df8b576c32/router/0.log" Jul 13 14:48:15.893002 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:15.892976 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5hbv_1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e/serve-healthcheck-canary/0.log" Jul 13 14:48:18.877034 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:18.876998 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6558cc6799-2pq6d" podUID="7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" containerName="console" containerID="cri-o://d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00" gracePeriod=15 Jul 13 14:48:19.117448 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.117421 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:48:19.183280 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183185 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-config\") pod \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " Jul 13 14:48:19.183280 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183257 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvwvr\" (UniqueName: \"kubernetes.io/projected/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-kube-api-access-rvwvr\") pod \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " Jul 13 14:48:19.183493 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183318 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-service-ca\") pod \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " Jul 13 14:48:19.183493 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183353 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-serving-cert\") pod \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " Jul 13 14:48:19.183493 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183390 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-oauth-config\") pod \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " Jul 13 14:48:19.183493 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183456 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-oauth-serving-cert\") pod \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\" (UID: \"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3\") " Jul 13 14:48:19.183700 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183610 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-config" (OuterVolumeSpecName: "console-config") pod "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" (UID: "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:48:19.183700 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183687 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-config\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:48:19.183814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183784 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-service-ca" (OuterVolumeSpecName: "service-ca") pod "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" (UID: "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:48:19.183927 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.183899 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" (UID: "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:48:19.185627 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.185596 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" (UID: "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:48:19.185739 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.185635 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-kube-api-access-rvwvr" (OuterVolumeSpecName: "kube-api-access-rvwvr") pod "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" (UID: "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3"). InnerVolumeSpecName "kube-api-access-rvwvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:48:19.185739 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.185634 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" (UID: "7228ecf9-0de3-4cb4-ab32-8e36de4fbce3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:48:19.284524 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.284481 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-oauth-serving-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:48:19.284524 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.284516 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwvr\" (UniqueName: \"kubernetes.io/projected/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-kube-api-access-rvwvr\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:48:19.284524 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.284526 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-service-ca\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:48:19.284777 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.284536 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-serving-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:48:19.284777 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.284545 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3-console-oauth-config\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:48:19.478559 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.478473 2571 generic.go:356] "Generic (PLEG): container finished" podID="7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" containerID="d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00" exitCode=0 Jul 13 14:48:19.478559 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.478511 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6558cc6799-2pq6d" Jul 13 14:48:19.478735 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.478552 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6558cc6799-2pq6d" event={"ID":"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3","Type":"ContainerDied","Data":"d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00"} Jul 13 14:48:19.478735 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.478606 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6558cc6799-2pq6d" event={"ID":"7228ecf9-0de3-4cb4-ab32-8e36de4fbce3","Type":"ContainerDied","Data":"5d866c97ab0731966eb868b7f2d9f1fd9f42e300ff2e3812258f6d3697994812"} Jul 13 14:48:19.478735 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.478627 2571 scope.go:117] "RemoveContainer" containerID="d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00" Jul 13 14:48:19.487862 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.487843 2571 scope.go:117] "RemoveContainer" containerID="d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00" Jul 13 14:48:19.488106 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:48:19.488086 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00\": container with ID starting with d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00 not found: ID does not exist" containerID="d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00" Jul 13 14:48:19.488146 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.488115 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00"} err="failed to get container status \"d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00\": rpc error: code = NotFound desc = could not find container \"d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00\": container with ID starting with d3f4238ccd6a5d44b67f216fb8a38d5202226dadd0b7486fca8b897ebb23ee00 not found: ID does not exist" Jul 13 14:48:19.503894 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.503866 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6558cc6799-2pq6d"] Jul 13 14:48:19.508532 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:19.508510 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6558cc6799-2pq6d"] Jul 13 14:48:20.868887 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:48:20.868851 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" path="/var/lib/kubelet/pods/7228ecf9-0de3-4cb4-ab32-8e36de4fbce3/volumes" Jul 13 14:49:14.725677 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.725638 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f7c59d57-jftcv"] Jul 13 14:49:14.726196 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.726023 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" containerName="console" Jul 13 14:49:14.726196 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.726038 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" containerName="console" Jul 13 14:49:14.726196 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.726098 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7228ecf9-0de3-4cb4-ab32-8e36de4fbce3" containerName="console" Jul 13 14:49:14.730451 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.730423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.733088 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.733062 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 14:49:14.733276 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.733241 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 14:49:14.734118 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.734097 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 14:49:14.734229 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.734101 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 14:49:14.734579 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.734559 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 14:49:14.734730 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.734710 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 14:49:14.734863 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.734774 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 14:49:14.734919 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.734897 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5hvf\"" Jul 13 14:49:14.739660 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.739639 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 14:49:14.741264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.741240 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7c59d57-jftcv"] Jul 13 14:49:14.821471 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.821428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-console-config\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.821471 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.821470 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgdbn\" (UniqueName: \"kubernetes.io/projected/e326671c-df0a-4235-aecd-69e3cc8579c4-kube-api-access-pgdbn\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.821720 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.821501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-trusted-ca-bundle\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.821720 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.821559 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-oauth-serving-cert\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.821720 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.821657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-service-ca\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.821720 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.821688 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-oauth-config\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.821720 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.821709 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-serving-cert\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923078 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-oauth-config\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923301 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923080 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-serving-cert\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923301 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-console-config\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923301 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgdbn\" (UniqueName: \"kubernetes.io/projected/e326671c-df0a-4235-aecd-69e3cc8579c4-kube-api-access-pgdbn\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923301 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923208 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-trusted-ca-bundle\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923301 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923240 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-oauth-serving-cert\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923549 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-service-ca\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.923929 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.923903 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-console-config\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.924068 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.924043 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-oauth-serving-cert\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.924124 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.924050 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-service-ca\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.924290 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.924269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-trusted-ca-bundle\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.925600 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.925579 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-oauth-config\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.925735 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.925717 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-serving-cert\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:14.932172 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:14.932130 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgdbn\" (UniqueName: \"kubernetes.io/projected/e326671c-df0a-4235-aecd-69e3cc8579c4-kube-api-access-pgdbn\") pod \"console-67f7c59d57-jftcv\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:15.041636 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:15.041598 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:15.172845 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:15.172816 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7c59d57-jftcv"] Jul 13 14:49:15.175568 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:49:15.175539 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode326671c_df0a_4235_aecd_69e3cc8579c4.slice/crio-c1944e2c7de2f1f5032351ffeaae389ac34e35efdd6696b0c7e076b1470e5405 WatchSource:0}: Error finding container c1944e2c7de2f1f5032351ffeaae389ac34e35efdd6696b0c7e076b1470e5405: Status 404 returned error can't find the container with id c1944e2c7de2f1f5032351ffeaae389ac34e35efdd6696b0c7e076b1470e5405 Jul 13 14:49:15.689025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:15.688991 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c59d57-jftcv" event={"ID":"e326671c-df0a-4235-aecd-69e3cc8579c4","Type":"ContainerStarted","Data":"21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8"} Jul 13 14:49:15.689218 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:15.689048 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c59d57-jftcv" event={"ID":"e326671c-df0a-4235-aecd-69e3cc8579c4","Type":"ContainerStarted","Data":"c1944e2c7de2f1f5032351ffeaae389ac34e35efdd6696b0c7e076b1470e5405"} Jul 13 14:49:15.709430 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:15.709358 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f7c59d57-jftcv" podStartSLOduration=1.7093407200000001 podStartE2EDuration="1.70934072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:49:15.708796009 +0000 UTC m=+193.362762477" watchObservedRunningTime="2026-07-13 14:49:15.70934072 +0000 UTC m=+193.363307175" Jul 13 14:49:25.042623 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:25.042583 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:25.043267 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:25.042657 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:25.047600 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:25.047574 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:49:25.727847 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:49:25.727817 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:50:45.029299 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.029265 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kdhsd"] Jul 13 14:50:45.032407 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.032384 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.034896 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.034869 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 13 14:50:45.035028 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.035010 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 13 14:50:45.035093 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.035081 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-czmk5\"" Jul 13 14:50:45.041458 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.041432 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kdhsd"] Jul 13 14:50:45.059915 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.059883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7t8\" (UniqueName: \"kubernetes.io/projected/ff5e95fa-7894-4390-904b-d0a91f1e8b5a-kube-api-access-wx7t8\") pod \"cert-manager-webhook-6486ffd997-kdhsd\" (UID: \"ff5e95fa-7894-4390-904b-d0a91f1e8b5a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.060079 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.059940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff5e95fa-7894-4390-904b-d0a91f1e8b5a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kdhsd\" (UID: \"ff5e95fa-7894-4390-904b-d0a91f1e8b5a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.160714 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.160679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff5e95fa-7894-4390-904b-d0a91f1e8b5a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kdhsd\" (UID: \"ff5e95fa-7894-4390-904b-d0a91f1e8b5a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.160864 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.160741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7t8\" (UniqueName: \"kubernetes.io/projected/ff5e95fa-7894-4390-904b-d0a91f1e8b5a-kube-api-access-wx7t8\") pod \"cert-manager-webhook-6486ffd997-kdhsd\" (UID: \"ff5e95fa-7894-4390-904b-d0a91f1e8b5a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.169781 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.169755 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff5e95fa-7894-4390-904b-d0a91f1e8b5a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kdhsd\" (UID: \"ff5e95fa-7894-4390-904b-d0a91f1e8b5a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.169902 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.169843 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7t8\" (UniqueName: \"kubernetes.io/projected/ff5e95fa-7894-4390-904b-d0a91f1e8b5a-kube-api-access-wx7t8\") pod \"cert-manager-webhook-6486ffd997-kdhsd\" (UID: \"ff5e95fa-7894-4390-904b-d0a91f1e8b5a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.343066 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.343028 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:50:45.474375 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:45.474346 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kdhsd"] Jul 13 14:50:45.477031 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:50:45.476990 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff5e95fa_7894_4390_904b_d0a91f1e8b5a.slice/crio-5cfee3509776ad0def9cda1f72879c741bc475d67be8670b89090f0daebb4ad2 WatchSource:0}: Error finding container 5cfee3509776ad0def9cda1f72879c741bc475d67be8670b89090f0daebb4ad2: Status 404 returned error can't find the container with id 5cfee3509776ad0def9cda1f72879c741bc475d67be8670b89090f0daebb4ad2 Jul 13 14:50:46.023679 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:50:46.023639 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" event={"ID":"ff5e95fa-7894-4390-904b-d0a91f1e8b5a","Type":"ContainerStarted","Data":"5cfee3509776ad0def9cda1f72879c741bc475d67be8670b89090f0daebb4ad2"} Jul 13 14:51:01.086794 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:01.086747 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" event={"ID":"ff5e95fa-7894-4390-904b-d0a91f1e8b5a","Type":"ContainerStarted","Data":"80ff2eaa0e30deda34640dd747a918f23d852a18216f64b82910c29d88ae5a8b"} Jul 13 14:51:01.087308 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:01.086869 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:51:01.105169 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:01.105093 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" podStartSLOduration=1.116020591 podStartE2EDuration="16.105078037s" totalImagesPullingTime="14.989057446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:51:01.103937193 +0000 UTC m=+298.757903646" watchObservedRunningTime="2026-07-13 14:51:01.105078037 +0000 UTC m=+298.759044490" Jul 13 14:51:02.771484 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:02.771448 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 14:51:02.771955 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:02.771769 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 14:51:02.781616 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:02.781562 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 14:51:07.095404 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:07.095375 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kdhsd" Jul 13 14:51:09.408693 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.408654 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42"] Jul 13 14:51:09.415783 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.415755 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.418447 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.418424 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 14:51:09.418565 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.418539 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-knj5m\"" Jul 13 14:51:09.418822 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.418806 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 14:51:09.418822 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.418815 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 14:51:09.418978 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.418960 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 14:51:09.423708 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.423687 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42"] Jul 13 14:51:09.463248 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.463210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd46cd42-458f-467b-bd6d-996cb935513c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.463437 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.463262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd46cd42-458f-467b-bd6d-996cb935513c-manifests\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.463437 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.463335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd46cd42-458f-467b-bd6d-996cb935513c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.463437 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.463362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khr2n\" (UniqueName: \"kubernetes.io/projected/dd46cd42-458f-467b-bd6d-996cb935513c-kube-api-access-khr2n\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.463560 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.463461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd46cd42-458f-467b-bd6d-996cb935513c-tmp\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.564285 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.564246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd46cd42-458f-467b-bd6d-996cb935513c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.564285 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.564287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khr2n\" (UniqueName: \"kubernetes.io/projected/dd46cd42-458f-467b-bd6d-996cb935513c-kube-api-access-khr2n\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.564527 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.564333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd46cd42-458f-467b-bd6d-996cb935513c-tmp\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.564527 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.564352 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd46cd42-458f-467b-bd6d-996cb935513c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.564527 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.564377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd46cd42-458f-467b-bd6d-996cb935513c-manifests\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.564839 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.564816 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd46cd42-458f-467b-bd6d-996cb935513c-manifests\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.566681 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.566651 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd46cd42-458f-467b-bd6d-996cb935513c-tmp\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.566787 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.566744 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd46cd42-458f-467b-bd6d-996cb935513c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.566889 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.566870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd46cd42-458f-467b-bd6d-996cb935513c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.574756 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.574735 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khr2n\" (UniqueName: \"kubernetes.io/projected/dd46cd42-458f-467b-bd6d-996cb935513c-kube-api-access-khr2n\") pod \"opendatahub-operator-controller-manager-6496967d77-f9z42\" (UID: \"dd46cd42-458f-467b-bd6d-996cb935513c\") " pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.727252 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.727135 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:09.870873 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.870843 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42"] Jul 13 14:51:09.873632 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:51:09.873585 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd46cd42_458f_467b_bd6d_996cb935513c.slice/crio-0b06980dc9573404eb3bfaa066834fd69257b8197b5f9f335f583255e339b999 WatchSource:0}: Error finding container 0b06980dc9573404eb3bfaa066834fd69257b8197b5f9f335f583255e339b999: Status 404 returned error can't find the container with id 0b06980dc9573404eb3bfaa066834fd69257b8197b5f9f335f583255e339b999 Jul 13 14:51:09.875413 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:09.875390 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:51:10.121640 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:10.121605 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" event={"ID":"dd46cd42-458f-467b-bd6d-996cb935513c","Type":"ContainerStarted","Data":"0b06980dc9573404eb3bfaa066834fd69257b8197b5f9f335f583255e339b999"} Jul 13 14:51:13.139404 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:13.139362 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" event={"ID":"dd46cd42-458f-467b-bd6d-996cb935513c","Type":"ContainerStarted","Data":"c339885fc2ec08a3630b013185b058b41014f63b04c2fad1282b737638d6f5b7"} Jul 13 14:51:15.148965 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:15.148929 2571 generic.go:356] "Generic (PLEG): container finished" podID="dd46cd42-458f-467b-bd6d-996cb935513c" containerID="c339885fc2ec08a3630b013185b058b41014f63b04c2fad1282b737638d6f5b7" exitCode=0 Jul 13 14:51:15.149341 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:15.148987 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" event={"ID":"dd46cd42-458f-467b-bd6d-996cb935513c","Type":"ContainerDied","Data":"c339885fc2ec08a3630b013185b058b41014f63b04c2fad1282b737638d6f5b7"} Jul 13 14:51:16.156018 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:16.155975 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" event={"ID":"dd46cd42-458f-467b-bd6d-996cb935513c","Type":"ContainerStarted","Data":"3bb06566a7559d836d0f842ae86aed2fd130f2face48165f75f2e7b5ed54375b"} Jul 13 14:51:16.156453 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:16.156333 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:16.181285 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:16.181221 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" podStartSLOduration=2.599967103 podStartE2EDuration="7.181201092s" totalImagesPullingTime="3.412146922s" totalInitContainerRuntime="1.169087067s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:51:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:51:16.178800171 +0000 UTC m=+313.832766624" watchObservedRunningTime="2026-07-13 14:51:16.181201092 +0000 UTC m=+313.835167551" Jul 13 14:51:27.162351 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:27.162319 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6496967d77-f9z42" Jul 13 14:51:30.901505 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.901476 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s"] Jul 13 14:51:30.904713 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.904697 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:30.907777 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.907744 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 14:51:30.909256 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.909240 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 14:51:30.921303 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.921275 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s"] Jul 13 14:51:30.958291 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.958255 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn5vt\" (UniqueName: \"kubernetes.io/projected/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-kube-api-access-vn5vt\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:30.958443 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.958423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-tls-certs\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:30.958484 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:30.958451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-tmp\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.058891 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.058850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn5vt\" (UniqueName: \"kubernetes.io/projected/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-kube-api-access-vn5vt\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.059084 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.059014 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-tls-certs\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.059084 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.059043 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-tmp\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.061671 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.061598 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-tmp\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.061941 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.061919 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-tls-certs\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.068631 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.068594 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn5vt\" (UniqueName: \"kubernetes.io/projected/39dc3323-01b4-4bf6-91a1-6d71d1daaeea-kube-api-access-vn5vt\") pod \"kube-auth-proxy-84c9577b7-gtd5s\" (UID: \"39dc3323-01b4-4bf6-91a1-6d71d1daaeea\") " pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.214064 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.213973 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" Jul 13 14:51:31.361170 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.361106 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fbjkf"] Jul 13 14:51:31.365375 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.365357 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:31.369639 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.369616 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-sdxhc\"" Jul 13 14:51:31.374368 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.374345 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 13 14:51:31.379671 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.379644 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s"] Jul 13 14:51:31.384339 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:51:31.384317 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39dc3323_01b4_4bf6_91a1_6d71d1daaeea.slice/crio-fd5ef3bfd6336f00bf26f37384b49e3c93eb9270567f2a7666843da2e5a5c99d WatchSource:0}: Error finding container fd5ef3bfd6336f00bf26f37384b49e3c93eb9270567f2a7666843da2e5a5c99d: Status 404 returned error can't find the container with id fd5ef3bfd6336f00bf26f37384b49e3c93eb9270567f2a7666843da2e5a5c99d Jul 13 14:51:31.391593 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.391568 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fbjkf"] Jul 13 14:51:31.462251 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.462212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhz72\" (UniqueName: \"kubernetes.io/projected/40821c7e-55eb-4aff-aed8-1a9fed459e5f-kube-api-access-zhz72\") pod \"odh-model-controller-7998c67dc4-fbjkf\" (UID: \"40821c7e-55eb-4aff-aed8-1a9fed459e5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:31.462426 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.462302 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40821c7e-55eb-4aff-aed8-1a9fed459e5f-cert\") pod \"odh-model-controller-7998c67dc4-fbjkf\" (UID: \"40821c7e-55eb-4aff-aed8-1a9fed459e5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:31.563264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.563216 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40821c7e-55eb-4aff-aed8-1a9fed459e5f-cert\") pod \"odh-model-controller-7998c67dc4-fbjkf\" (UID: \"40821c7e-55eb-4aff-aed8-1a9fed459e5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:31.563472 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.563325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhz72\" (UniqueName: \"kubernetes.io/projected/40821c7e-55eb-4aff-aed8-1a9fed459e5f-kube-api-access-zhz72\") pod \"odh-model-controller-7998c67dc4-fbjkf\" (UID: \"40821c7e-55eb-4aff-aed8-1a9fed459e5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:31.563472 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:31.563391 2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 13 14:51:31.563591 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:31.563474 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40821c7e-55eb-4aff-aed8-1a9fed459e5f-cert podName:40821c7e-55eb-4aff-aed8-1a9fed459e5f nodeName:}" failed. No retries permitted until 2026-07-13 14:51:32.063451677 +0000 UTC m=+329.717418126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/40821c7e-55eb-4aff-aed8-1a9fed459e5f-cert") pod "odh-model-controller-7998c67dc4-fbjkf" (UID: "40821c7e-55eb-4aff-aed8-1a9fed459e5f") : secret "odh-model-controller-webhook-cert" not found Jul 13 14:51:31.573908 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:31.573872 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhz72\" (UniqueName: \"kubernetes.io/projected/40821c7e-55eb-4aff-aed8-1a9fed459e5f-kube-api-access-zhz72\") pod \"odh-model-controller-7998c67dc4-fbjkf\" (UID: \"40821c7e-55eb-4aff-aed8-1a9fed459e5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:32.067419 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.067383 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40821c7e-55eb-4aff-aed8-1a9fed459e5f-cert\") pod \"odh-model-controller-7998c67dc4-fbjkf\" (UID: \"40821c7e-55eb-4aff-aed8-1a9fed459e5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:32.069828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.069800 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40821c7e-55eb-4aff-aed8-1a9fed459e5f-cert\") pod \"odh-model-controller-7998c67dc4-fbjkf\" (UID: \"40821c7e-55eb-4aff-aed8-1a9fed459e5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:32.229425 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.229380 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" event={"ID":"39dc3323-01b4-4bf6-91a1-6d71d1daaeea","Type":"ContainerStarted","Data":"fd5ef3bfd6336f00bf26f37384b49e3c93eb9270567f2a7666843da2e5a5c99d"} Jul 13 14:51:32.277251 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.277137 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:32.454640 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.453302 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rz478"] Jul 13 14:51:32.458783 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.457863 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fbjkf"] Jul 13 14:51:32.458783 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.457985 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.463063 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.462092 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 13 14:51:32.463063 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.462450 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 13 14:51:32.463063 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.462613 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 13 14:51:32.463063 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.462792 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8wjbt\"" Jul 13 14:51:32.463063 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.462946 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 14:51:32.466734 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:51:32.466220 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40821c7e_55eb_4aff_aed8_1a9fed459e5f.slice/crio-2ccb115aad2eaa155b7fccbd762727d4c780ea31f8647f54d0888cb784114bce WatchSource:0}: Error finding container 2ccb115aad2eaa155b7fccbd762727d4c780ea31f8647f54d0888cb784114bce: Status 404 returned error can't find the container with id 2ccb115aad2eaa155b7fccbd762727d4c780ea31f8647f54d0888cb784114bce Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.472904 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rz478"] Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.473301 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-istio-token\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.473342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.473388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvwm\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-kube-api-access-4gvwm\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.473427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.473476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3977014-48fe-4a09-b704-ac85853d6091-local-certs\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.473507 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-cacerts\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.474444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.473537 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.574372 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.574288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-istio-token\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.574372 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.574356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.574627 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.574419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvwm\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-kube-api-access-4gvwm\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.574627 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.574463 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.574627 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.574515 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3977014-48fe-4a09-b704-ac85853d6091-local-certs\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.574627 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.574548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-cacerts\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.574627 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.574585 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.575400 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.575312 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.577569 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.577508 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.577797 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.577731 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3977014-48fe-4a09-b704-ac85853d6091-local-certs\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.578069 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.578041 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.579393 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.579368 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-cacerts\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.585070 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.585039 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvwm\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-kube-api-access-4gvwm\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.585212 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.585111 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-istio-token\") pod \"istiod-openshift-gateway-7574459656-rz478\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.785998 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.785970 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:32.937391 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:32.937349 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rz478"] Jul 13 14:51:32.939467 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:51:32.939439 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3977014_48fe_4a09_b704_ac85853d6091.slice/crio-df07e1a0ceb3630c8e7e01c97698923c31614b34740fc0c58f85a40b53af732c WatchSource:0}: Error finding container df07e1a0ceb3630c8e7e01c97698923c31614b34740fc0c58f85a40b53af732c: Status 404 returned error can't find the container with id df07e1a0ceb3630c8e7e01c97698923c31614b34740fc0c58f85a40b53af732c Jul 13 14:51:33.235717 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:33.235623 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" event={"ID":"c3977014-48fe-4a09-b704-ac85853d6091","Type":"ContainerStarted","Data":"df07e1a0ceb3630c8e7e01c97698923c31614b34740fc0c58f85a40b53af732c"} Jul 13 14:51:33.237174 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:33.237129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" event={"ID":"40821c7e-55eb-4aff-aed8-1a9fed459e5f","Type":"ContainerStarted","Data":"2ccb115aad2eaa155b7fccbd762727d4c780ea31f8647f54d0888cb784114bce"} Jul 13 14:51:35.253193 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:35.252219 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" event={"ID":"39dc3323-01b4-4bf6-91a1-6d71d1daaeea","Type":"ContainerStarted","Data":"43ee282ed3d0d35b56fad892d9c5e6dabcca6c995ff35a0a9141bc463509ab84"} Jul 13 14:51:35.272647 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:35.272561 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84c9577b7-gtd5s" podStartSLOduration=1.958824892 podStartE2EDuration="5.272540598s" totalImagesPullingTime="3.313715706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:51:35.270626582 +0000 UTC m=+332.924593037" watchObservedRunningTime="2026-07-13 14:51:35.272540598 +0000 UTC m=+332.926507053" Jul 13 14:51:37.544790 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:37.544753 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:51:37.545187 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:37.544835 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:51:38.271283 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.271234 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" event={"ID":"40821c7e-55eb-4aff-aed8-1a9fed459e5f","Type":"ContainerStarted","Data":"760e1802ccb556cbefd77ec725f272350b74dbd0417e99beaf8746edecb307de"} Jul 13 14:51:38.271480 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.271410 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:38.273142 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.273114 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" event={"ID":"c3977014-48fe-4a09-b704-ac85853d6091","Type":"ContainerStarted","Data":"c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8"} Jul 13 14:51:38.273274 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.273187 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:38.275026 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.274998 2571 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-rz478 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 14:51:38.275126 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.275050 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" podUID="c3977014-48fe-4a09-b704-ac85853d6091" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 14:51:38.300369 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.300316 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" podStartSLOduration=2.224832233 podStartE2EDuration="7.300297815s" totalImagesPullingTime="5.075465582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:51:38.299783085 +0000 UTC m=+335.953749540" watchObservedRunningTime="2026-07-13 14:51:38.300297815 +0000 UTC m=+335.954264272" Jul 13 14:51:38.325322 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:38.325264 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" podStartSLOduration=1.723483611 podStartE2EDuration="6.325246129s" totalImagesPullingTime="4.601762518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:51:38.323568035 +0000 UTC m=+335.977534490" watchObservedRunningTime="2026-07-13 14:51:38.325246129 +0000 UTC m=+335.979212583" Jul 13 14:51:39.280555 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:39.280340 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:42.667323 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.667288 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4"] Jul 13 14:51:42.671130 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.671093 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:42.673745 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.673720 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt"] Jul 13 14:51:42.674241 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.674220 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lpm4c\"" Jul 13 14:51:42.674571 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.674542 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 13 14:51:42.677126 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.677105 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:42.680761 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.680735 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 14:51:42.681048 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.681030 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ltk4x\"" Jul 13 14:51:42.683415 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.683392 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4"] Jul 13 14:51:42.686450 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.686429 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt"] Jul 13 14:51:42.775301 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.775206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2ns\" (UniqueName: \"kubernetes.io/projected/591cb838-631d-47f7-aad4-fe2b3f0f6d81-kube-api-access-2g2ns\") pod \"kserve-controller-manager-d6f4b65b8-tkrt4\" (UID: \"591cb838-631d-47f7-aad4-fe2b3f0f6d81\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:42.775301 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.775258 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hvn\" (UniqueName: \"kubernetes.io/projected/291f8a43-888a-4e3b-92a9-a6a04c6812d1-kube-api-access-z5hvn\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:42.775520 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.775335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/591cb838-631d-47f7-aad4-fe2b3f0f6d81-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrt4\" (UID: \"591cb838-631d-47f7-aad4-fe2b3f0f6d81\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:42.775520 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.775417 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.876585 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2ns\" (UniqueName: \"kubernetes.io/projected/591cb838-631d-47f7-aad4-fe2b3f0f6d81-kube-api-access-2g2ns\") pod \"kserve-controller-manager-d6f4b65b8-tkrt4\" (UID: \"591cb838-631d-47f7-aad4-fe2b3f0f6d81\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.876645 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hvn\" (UniqueName: \"kubernetes.io/projected/291f8a43-888a-4e3b-92a9-a6a04c6812d1-kube-api-access-z5hvn\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.876678 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/591cb838-631d-47f7-aad4-fe2b3f0f6d81-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrt4\" (UID: \"591cb838-631d-47f7-aad4-fe2b3f0f6d81\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.876737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:42.876853 2571 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:42.876915 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert podName:291f8a43-888a-4e3b-92a9-a6a04c6812d1 nodeName:}" failed. No retries permitted until 2026-07-13 14:51:43.376893393 +0000 UTC m=+341.030859828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert") pod "llmisvc-controller-manager-777c6bd7d5-dh7zt" (UID: "291f8a43-888a-4e3b-92a9-a6a04c6812d1") : secret "llmisvc-webhook-server-cert" not found Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:42.877053 2571 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 13 14:51:42.878177 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:42.877108 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591cb838-631d-47f7-aad4-fe2b3f0f6d81-cert podName:591cb838-631d-47f7-aad4-fe2b3f0f6d81 nodeName:}" failed. No retries permitted until 2026-07-13 14:51:43.377091224 +0000 UTC m=+341.031057658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/591cb838-631d-47f7-aad4-fe2b3f0f6d81-cert") pod "kserve-controller-manager-d6f4b65b8-tkrt4" (UID: "591cb838-631d-47f7-aad4-fe2b3f0f6d81") : secret "kserve-webhook-server-cert" not found Jul 13 14:51:42.894088 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.894055 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2ns\" (UniqueName: \"kubernetes.io/projected/591cb838-631d-47f7-aad4-fe2b3f0f6d81-kube-api-access-2g2ns\") pod \"kserve-controller-manager-d6f4b65b8-tkrt4\" (UID: \"591cb838-631d-47f7-aad4-fe2b3f0f6d81\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:42.895072 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:42.895040 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hvn\" (UniqueName: \"kubernetes.io/projected/291f8a43-888a-4e3b-92a9-a6a04c6812d1-kube-api-access-z5hvn\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:43.382468 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:43.382427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:43.382681 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:43.382546 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/591cb838-631d-47f7-aad4-fe2b3f0f6d81-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrt4\" (UID: \"591cb838-631d-47f7-aad4-fe2b3f0f6d81\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:43.382681 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:43.382572 2571 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 13 14:51:43.382681 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:43.382644 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert podName:291f8a43-888a-4e3b-92a9-a6a04c6812d1 nodeName:}" failed. No retries permitted until 2026-07-13 14:51:44.382626395 +0000 UTC m=+342.036592828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert") pod "llmisvc-controller-manager-777c6bd7d5-dh7zt" (UID: "291f8a43-888a-4e3b-92a9-a6a04c6812d1") : secret "llmisvc-webhook-server-cert" not found Jul 13 14:51:43.385019 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:43.384999 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/591cb838-631d-47f7-aad4-fe2b3f0f6d81-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrt4\" (UID: \"591cb838-631d-47f7-aad4-fe2b3f0f6d81\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:43.588638 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:43.588609 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:43.733620 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:43.730996 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4"] Jul 13 14:51:43.878640 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:43.878601 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rz478"] Jul 13 14:51:43.878942 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:43.878853 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" podUID="c3977014-48fe-4a09-b704-ac85853d6091" containerName="discovery" containerID="cri-o://c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8" gracePeriod=30 Jul 13 14:51:44.126011 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.125985 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:44.190871 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.190792 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-cacerts\") pod \"c3977014-48fe-4a09-b704-ac85853d6091\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " Jul 13 14:51:44.190871 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.190840 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-kubeconfig\") pod \"c3977014-48fe-4a09-b704-ac85853d6091\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " Jul 13 14:51:44.191060 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.190884 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gvwm\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-kube-api-access-4gvwm\") pod \"c3977014-48fe-4a09-b704-ac85853d6091\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " Jul 13 14:51:44.191060 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.190916 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-istio-token\") pod \"c3977014-48fe-4a09-b704-ac85853d6091\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " Jul 13 14:51:44.191060 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.190941 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-ca-configmap\") pod \"c3977014-48fe-4a09-b704-ac85853d6091\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " Jul 13 14:51:44.191060 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.191003 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-dns-cert\") pod \"c3977014-48fe-4a09-b704-ac85853d6091\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " Jul 13 14:51:44.191060 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.191026 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3977014-48fe-4a09-b704-ac85853d6091-local-certs\") pod \"c3977014-48fe-4a09-b704-ac85853d6091\" (UID: \"c3977014-48fe-4a09-b704-ac85853d6091\") " Jul 13 14:51:44.192573 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.192523 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c3977014-48fe-4a09-b704-ac85853d6091" (UID: "c3977014-48fe-4a09-b704-ac85853d6091"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:51:44.193370 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.193339 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-cacerts" (OuterVolumeSpecName: "cacerts") pod "c3977014-48fe-4a09-b704-ac85853d6091" (UID: "c3977014-48fe-4a09-b704-ac85853d6091"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:51:44.194399 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.194362 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3977014-48fe-4a09-b704-ac85853d6091-local-certs" (OuterVolumeSpecName: "local-certs") pod "c3977014-48fe-4a09-b704-ac85853d6091" (UID: "c3977014-48fe-4a09-b704-ac85853d6091"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:51:44.194500 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.194413 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-kube-api-access-4gvwm" (OuterVolumeSpecName: "kube-api-access-4gvwm") pod "c3977014-48fe-4a09-b704-ac85853d6091" (UID: "c3977014-48fe-4a09-b704-ac85853d6091"). InnerVolumeSpecName "kube-api-access-4gvwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:51:44.194544 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.194501 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c3977014-48fe-4a09-b704-ac85853d6091" (UID: "c3977014-48fe-4a09-b704-ac85853d6091"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:51:44.194634 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.194615 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3977014-48fe-4a09-b704-ac85853d6091" (UID: "c3977014-48fe-4a09-b704-ac85853d6091"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:51:44.194696 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.194637 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c3977014-48fe-4a09-b704-ac85853d6091" (UID: "c3977014-48fe-4a09-b704-ac85853d6091"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:51:44.291970 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.291931 2571 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-dns-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:51:44.291970 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.291962 2571 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3977014-48fe-4a09-b704-ac85853d6091-local-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:51:44.291970 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.291971 2571 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-cacerts\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:51:44.291970 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.291979 2571 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3977014-48fe-4a09-b704-ac85853d6091-istio-kubeconfig\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:51:44.292344 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.291987 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvwm\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-kube-api-access-4gvwm\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:51:44.292344 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.291997 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3977014-48fe-4a09-b704-ac85853d6091-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:51:44.292344 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.292005 2571 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3977014-48fe-4a09-b704-ac85853d6091-istio-csr-ca-configmap\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:51:44.300083 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.300043 2571 generic.go:356] "Generic (PLEG): container finished" podID="c3977014-48fe-4a09-b704-ac85853d6091" containerID="c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8" exitCode=0 Jul 13 14:51:44.300286 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.300093 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" Jul 13 14:51:44.300286 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.300129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" event={"ID":"c3977014-48fe-4a09-b704-ac85853d6091","Type":"ContainerDied","Data":"c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8"} Jul 13 14:51:44.300286 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.300214 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rz478" event={"ID":"c3977014-48fe-4a09-b704-ac85853d6091","Type":"ContainerDied","Data":"df07e1a0ceb3630c8e7e01c97698923c31614b34740fc0c58f85a40b53af732c"} Jul 13 14:51:44.300286 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.300243 2571 scope.go:117] "RemoveContainer" containerID="c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8" Jul 13 14:51:44.301630 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.301601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" event={"ID":"591cb838-631d-47f7-aad4-fe2b3f0f6d81","Type":"ContainerStarted","Data":"5d3e5e544433d3a5ce933625ea4c827987ed2e2aee6614296fa5a03eb96ab3b7"} Jul 13 14:51:44.310509 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.310492 2571 scope.go:117] "RemoveContainer" containerID="c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8" Jul 13 14:51:44.310752 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:51:44.310733 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8\": container with ID starting with c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8 not found: ID does not exist" containerID="c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8" Jul 13 14:51:44.310807 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.310760 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8"} err="failed to get container status \"c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8\": rpc error: code = NotFound desc = could not find container \"c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8\": container with ID starting with c4993297a6677ff519f0422d1cab292a741b3633f389e4a31fe1da0366077eb8 not found: ID does not exist" Jul 13 14:51:44.324284 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.324256 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rz478"] Jul 13 14:51:44.328437 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.328413 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rz478"] Jul 13 14:51:44.393146 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.393109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:44.395578 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.395554 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/291f8a43-888a-4e3b-92a9-a6a04c6812d1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dh7zt\" (UID: \"291f8a43-888a-4e3b-92a9-a6a04c6812d1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:44.495171 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.495070 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:44.631227 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.631031 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt"] Jul 13 14:51:44.633579 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:51:44.633544 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod291f8a43_888a_4e3b_92a9_a6a04c6812d1.slice/crio-29497bb72616dbd2201c8cd34c1c044105ca10ac7298c3fe0a4785d6acd874d7 WatchSource:0}: Error finding container 29497bb72616dbd2201c8cd34c1c044105ca10ac7298c3fe0a4785d6acd874d7: Status 404 returned error can't find the container with id 29497bb72616dbd2201c8cd34c1c044105ca10ac7298c3fe0a4785d6acd874d7 Jul 13 14:51:44.869606 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:44.869563 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3977014-48fe-4a09-b704-ac85853d6091" path="/var/lib/kubelet/pods/c3977014-48fe-4a09-b704-ac85853d6091/volumes" Jul 13 14:51:45.308232 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:45.308194 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" event={"ID":"291f8a43-888a-4e3b-92a9-a6a04c6812d1","Type":"ContainerStarted","Data":"29497bb72616dbd2201c8cd34c1c044105ca10ac7298c3fe0a4785d6acd874d7"} Jul 13 14:51:47.320020 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:47.319990 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" event={"ID":"591cb838-631d-47f7-aad4-fe2b3f0f6d81","Type":"ContainerStarted","Data":"f3dbb3a0a22e2be82dab433444316a1a581542d321cf4649e2332f91cc68d5be"} Jul 13 14:51:47.320483 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:47.320062 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:51:47.336680 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:47.336551 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" podStartSLOduration=1.9988815039999999 podStartE2EDuration="5.336504397s" totalImagesPullingTime="3.337622893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:51:47.335985341 +0000 UTC m=+344.989951809" watchObservedRunningTime="2026-07-13 14:51:47.336504397 +0000 UTC m=+344.990470852" Jul 13 14:51:49.280726 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:49.280697 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fbjkf" Jul 13 14:51:50.336626 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:50.336579 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" event={"ID":"291f8a43-888a-4e3b-92a9-a6a04c6812d1","Type":"ContainerStarted","Data":"d4f18ba0a42d18c0f07e07a5b177639967289270aafc1a99834f9c3d7249d13b"} Jul 13 14:51:50.337070 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:50.336769 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:51:50.355674 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:51:50.355626 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" podStartSLOduration=2.985223397 podStartE2EDuration="8.355610191s" totalImagesPullingTime="5.370386794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:51:50.353847887 +0000 UTC m=+348.007814357" watchObservedRunningTime="2026-07-13 14:51:50.355610191 +0000 UTC m=+348.009576644" Jul 13 14:52:18.330555 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:18.330524 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrt4" Jul 13 14:52:21.344662 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:21.344631 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dh7zt" Jul 13 14:52:32.212408 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.212371 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q94lv"] Jul 13 14:52:32.212834 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.212767 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3977014-48fe-4a09-b704-ac85853d6091" containerName="discovery" Jul 13 14:52:32.212834 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.212781 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3977014-48fe-4a09-b704-ac85853d6091" containerName="discovery" Jul 13 14:52:32.212917 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.212853 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3977014-48fe-4a09-b704-ac85853d6091" containerName="discovery" Jul 13 14:52:32.215865 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.215840 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" Jul 13 14:52:32.218840 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.218817 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 14:52:32.219814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.219792 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6p8q5\"" Jul 13 14:52:32.219939 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.219813 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 14:52:32.227227 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.227203 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q94lv"] Jul 13 14:52:32.298030 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.297993 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jd9c\" (UniqueName: \"kubernetes.io/projected/a28eb6f9-1c4f-4431-9be0-50e98cbc4dca-kube-api-access-7jd9c\") pod \"kuadrant-operator-catalog-q94lv\" (UID: \"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca\") " pod="kuadrant-system/kuadrant-operator-catalog-q94lv" Jul 13 14:52:32.398600 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.398558 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jd9c\" (UniqueName: \"kubernetes.io/projected/a28eb6f9-1c4f-4431-9be0-50e98cbc4dca-kube-api-access-7jd9c\") pod \"kuadrant-operator-catalog-q94lv\" (UID: \"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca\") " pod="kuadrant-system/kuadrant-operator-catalog-q94lv" Jul 13 14:52:32.407430 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.407402 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jd9c\" (UniqueName: \"kubernetes.io/projected/a28eb6f9-1c4f-4431-9be0-50e98cbc4dca-kube-api-access-7jd9c\") pod \"kuadrant-operator-catalog-q94lv\" (UID: \"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca\") " pod="kuadrant-system/kuadrant-operator-catalog-q94lv" Jul 13 14:52:32.528087 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.528005 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" Jul 13 14:52:32.578894 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.578864 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q94lv"] Jul 13 14:52:32.659606 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.659579 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q94lv"] Jul 13 14:52:32.662286 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:52:32.662260 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28eb6f9_1c4f_4431_9be0_50e98cbc4dca.slice/crio-cf2af6425de2180f0de49c1d8583a978208b20326e1868b40a0b3da695eac8ee WatchSource:0}: Error finding container cf2af6425de2180f0de49c1d8583a978208b20326e1868b40a0b3da695eac8ee: Status 404 returned error can't find the container with id cf2af6425de2180f0de49c1d8583a978208b20326e1868b40a0b3da695eac8ee Jul 13 14:52:32.787328 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.787290 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v5dd6"] Jul 13 14:52:32.791632 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.791608 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:32.797710 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.797686 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v5dd6"] Jul 13 14:52:32.902855 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:32.902552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dk8g\" (UniqueName: \"kubernetes.io/projected/25bb007c-a58b-488e-9fbe-cdb74c52b766-kube-api-access-9dk8g\") pod \"kuadrant-operator-catalog-v5dd6\" (UID: \"25bb007c-a58b-488e-9fbe-cdb74c52b766\") " pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:33.003184 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:33.003136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dk8g\" (UniqueName: \"kubernetes.io/projected/25bb007c-a58b-488e-9fbe-cdb74c52b766-kube-api-access-9dk8g\") pod \"kuadrant-operator-catalog-v5dd6\" (UID: \"25bb007c-a58b-488e-9fbe-cdb74c52b766\") " pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:33.012370 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:33.012335 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dk8g\" (UniqueName: \"kubernetes.io/projected/25bb007c-a58b-488e-9fbe-cdb74c52b766-kube-api-access-9dk8g\") pod \"kuadrant-operator-catalog-v5dd6\" (UID: \"25bb007c-a58b-488e-9fbe-cdb74c52b766\") " pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:33.102397 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:33.102316 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:33.241771 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:33.241744 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v5dd6"] Jul 13 14:52:33.243396 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:52:33.243374 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bb007c_a58b_488e_9fbe_cdb74c52b766.slice/crio-c52a4276f1b9e8b1f6cff11061e84800e3eb69a9af329c8da928cfdfe3508fbc WatchSource:0}: Error finding container c52a4276f1b9e8b1f6cff11061e84800e3eb69a9af329c8da928cfdfe3508fbc: Status 404 returned error can't find the container with id c52a4276f1b9e8b1f6cff11061e84800e3eb69a9af329c8da928cfdfe3508fbc Jul 13 14:52:33.527208 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:33.527109 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" event={"ID":"25bb007c-a58b-488e-9fbe-cdb74c52b766","Type":"ContainerStarted","Data":"c52a4276f1b9e8b1f6cff11061e84800e3eb69a9af329c8da928cfdfe3508fbc"} Jul 13 14:52:33.528312 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:33.528286 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" event={"ID":"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca","Type":"ContainerStarted","Data":"cf2af6425de2180f0de49c1d8583a978208b20326e1868b40a0b3da695eac8ee"} Jul 13 14:52:36.546889 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.546850 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" event={"ID":"25bb007c-a58b-488e-9fbe-cdb74c52b766","Type":"ContainerStarted","Data":"7d99ecc32a88960c27cd549a2ec103661ed37a8d391661fc815c1fb2b390e257"} Jul 13 14:52:36.548332 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.548309 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" event={"ID":"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca","Type":"ContainerStarted","Data":"0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300"} Jul 13 14:52:36.548411 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.548381 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" podUID="a28eb6f9-1c4f-4431-9be0-50e98cbc4dca" containerName="registry-server" containerID="cri-o://0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300" gracePeriod=2 Jul 13 14:52:36.565118 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.565073 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" podStartSLOduration=2.232559951 podStartE2EDuration="4.565058369s" totalImagesPullingTime="2.332498418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:36.564043912 +0000 UTC m=+394.218010368" watchObservedRunningTime="2026-07-13 14:52:36.565058369 +0000 UTC m=+394.219024823" Jul 13 14:52:36.581386 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.581343 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" podStartSLOduration=1.664524176 podStartE2EDuration="4.581328126s" totalImagesPullingTime="2.91680395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:36.579874697 +0000 UTC m=+394.233841151" watchObservedRunningTime="2026-07-13 14:52:36.581328126 +0000 UTC m=+394.235294582" Jul 13 14:52:36.788693 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.788666 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" Jul 13 14:52:36.837368 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.837300 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jd9c\" (UniqueName: \"kubernetes.io/projected/a28eb6f9-1c4f-4431-9be0-50e98cbc4dca-kube-api-access-7jd9c\") pod \"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca\" (UID: \"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca\") " Jul 13 14:52:36.839347 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.839324 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28eb6f9-1c4f-4431-9be0-50e98cbc4dca-kube-api-access-7jd9c" (OuterVolumeSpecName: "kube-api-access-7jd9c") pod "a28eb6f9-1c4f-4431-9be0-50e98cbc4dca" (UID: "a28eb6f9-1c4f-4431-9be0-50e98cbc4dca"). InnerVolumeSpecName "kube-api-access-7jd9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:52:36.938361 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:36.938333 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jd9c\" (UniqueName: \"kubernetes.io/projected/a28eb6f9-1c4f-4431-9be0-50e98cbc4dca-kube-api-access-7jd9c\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:52:37.554180 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.554120 2571 generic.go:356] "Generic (PLEG): container finished" podID="a28eb6f9-1c4f-4431-9be0-50e98cbc4dca" containerID="0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300" exitCode=0 Jul 13 14:52:37.554658 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.554186 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" Jul 13 14:52:37.554658 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.554204 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" event={"ID":"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca","Type":"ContainerDied","Data":"0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300"} Jul 13 14:52:37.554658 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.554259 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q94lv" event={"ID":"a28eb6f9-1c4f-4431-9be0-50e98cbc4dca","Type":"ContainerDied","Data":"cf2af6425de2180f0de49c1d8583a978208b20326e1868b40a0b3da695eac8ee"} Jul 13 14:52:37.554658 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.554280 2571 scope.go:117] "RemoveContainer" containerID="0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300" Jul 13 14:52:37.562833 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.562816 2571 scope.go:117] "RemoveContainer" containerID="0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300" Jul 13 14:52:37.563098 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:52:37.563072 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300\": container with ID starting with 0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300 not found: ID does not exist" containerID="0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300" Jul 13 14:52:37.563142 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.563107 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300"} err="failed to get container status \"0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300\": rpc error: code = NotFound desc = could not find container \"0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300\": container with ID starting with 0df0801d5e050f686fc1401b256d8a0de9bc865a2a61b6ec8608ce9b14d89300 not found: ID does not exist" Jul 13 14:52:37.572817 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.572790 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q94lv"] Jul 13 14:52:37.576805 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:37.576785 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q94lv"] Jul 13 14:52:38.869124 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:38.869093 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28eb6f9-1c4f-4431-9be0-50e98cbc4dca" path="/var/lib/kubelet/pods/a28eb6f9-1c4f-4431-9be0-50e98cbc4dca/volumes" Jul 13 14:52:43.103039 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:43.102999 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:43.103482 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:43.103058 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:43.125625 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:43.125598 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:52:43.605017 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:52:43.604988 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-v5dd6" Jul 13 14:53:02.544480 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.544446 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d65f5fdf-qbqcw"] Jul 13 14:53:02.545099 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.545060 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28eb6f9-1c4f-4431-9be0-50e98cbc4dca" containerName="registry-server" Jul 13 14:53:02.545099 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.545091 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28eb6f9-1c4f-4431-9be0-50e98cbc4dca" containerName="registry-server" Jul 13 14:53:02.545375 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.545195 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28eb6f9-1c4f-4431-9be0-50e98cbc4dca" containerName="registry-server" Jul 13 14:53:02.553187 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.553143 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.557999 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.557967 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d65f5fdf-qbqcw"] Jul 13 14:53:02.661831 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.661797 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-trusted-ca-bundle\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.661997 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.661842 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-service-ca\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.661997 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.661902 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-config\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.661997 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.661926 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-oauth-serving-cert\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.661997 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.661951 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5zp\" (UniqueName: \"kubernetes.io/projected/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-kube-api-access-6x5zp\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.662175 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.662004 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-serving-cert\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.662175 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.662023 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-oauth-config\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.762548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.762521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-trusted-ca-bundle\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.762732 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.762556 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-service-ca\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.762732 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.762586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-config\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.762732 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.762604 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-oauth-serving-cert\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.762732 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.762705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5zp\" (UniqueName: \"kubernetes.io/projected/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-kube-api-access-6x5zp\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.763000 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.762773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-serving-cert\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.763000 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.762802 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-oauth-config\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.763348 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.763323 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-oauth-serving-cert\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.763466 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.763357 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-config\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.763466 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.763369 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-service-ca\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.763555 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.763541 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-trusted-ca-bundle\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.765324 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.765305 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-oauth-config\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.765404 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.765376 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-console-serving-cert\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.771333 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.771306 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5zp\" (UniqueName: \"kubernetes.io/projected/7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06-kube-api-access-6x5zp\") pod \"console-67d65f5fdf-qbqcw\" (UID: \"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06\") " pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:02.863744 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:02.863717 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:03.020045 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.020006 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d65f5fdf-qbqcw"] Jul 13 14:53:03.022467 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:03.022440 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddd9b3d_0818_49d3_8d8f_c8d83fe1af06.slice/crio-0e096263f6d566e7c9809537bcaf68cdb90ec8577bf7933c619bb31edd7a444c WatchSource:0}: Error finding container 0e096263f6d566e7c9809537bcaf68cdb90ec8577bf7933c619bb31edd7a444c: Status 404 returned error can't find the container with id 0e096263f6d566e7c9809537bcaf68cdb90ec8577bf7933c619bb31edd7a444c Jul 13 14:53:03.675447 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.675415 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d65f5fdf-qbqcw" event={"ID":"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06","Type":"ContainerStarted","Data":"9020ce15ba632996401bd7301df4789eb8b52739515fc5f93c68176b226fa5c0"} Jul 13 14:53:03.675822 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.675463 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d65f5fdf-qbqcw" event={"ID":"7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06","Type":"ContainerStarted","Data":"0e096263f6d566e7c9809537bcaf68cdb90ec8577bf7933c619bb31edd7a444c"} Jul 13 14:53:03.692191 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.692140 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx"] Jul 13 14:53:03.695413 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.695391 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" Jul 13 14:53:03.699282 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.699257 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 14:53:03.699417 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.699309 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v422s\"" Jul 13 14:53:03.716102 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.716071 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx"] Jul 13 14:53:03.718230 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.718175 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67d65f5fdf-qbqcw" podStartSLOduration=1.718146458 podStartE2EDuration="1.718146458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:03.716617209 +0000 UTC m=+421.370583663" watchObservedRunningTime="2026-07-13 14:53:03.718146458 +0000 UTC m=+421.372112912" Jul 13 14:53:03.873203 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.873145 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mddb\" (UniqueName: \"kubernetes.io/projected/d886f0d5-ada3-4918-8536-03fd6a632ed3-kube-api-access-8mddb\") pod \"dns-operator-controller-manager-864dd78c4c-c9lkx\" (UID: \"d886f0d5-ada3-4918-8536-03fd6a632ed3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" Jul 13 14:53:03.974735 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.974595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mddb\" (UniqueName: \"kubernetes.io/projected/d886f0d5-ada3-4918-8536-03fd6a632ed3-kube-api-access-8mddb\") pod \"dns-operator-controller-manager-864dd78c4c-c9lkx\" (UID: \"d886f0d5-ada3-4918-8536-03fd6a632ed3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" Jul 13 14:53:03.986206 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:03.986163 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mddb\" (UniqueName: \"kubernetes.io/projected/d886f0d5-ada3-4918-8536-03fd6a632ed3-kube-api-access-8mddb\") pod \"dns-operator-controller-manager-864dd78c4c-c9lkx\" (UID: \"d886f0d5-ada3-4918-8536-03fd6a632ed3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" Jul 13 14:53:04.005331 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:04.005297 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" Jul 13 14:53:04.165032 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:04.164977 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd886f0d5_ada3_4918_8536_03fd6a632ed3.slice/crio-c2adf75c38901fdbeded99c6efafacc35fa406afc00029ebfc6d4488762b2e95 WatchSource:0}: Error finding container c2adf75c38901fdbeded99c6efafacc35fa406afc00029ebfc6d4488762b2e95: Status 404 returned error can't find the container with id c2adf75c38901fdbeded99c6efafacc35fa406afc00029ebfc6d4488762b2e95 Jul 13 14:53:04.175280 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:04.175251 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx"] Jul 13 14:53:04.681303 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:04.681258 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" event={"ID":"d886f0d5-ada3-4918-8536-03fd6a632ed3","Type":"ContainerStarted","Data":"c2adf75c38901fdbeded99c6efafacc35fa406afc00029ebfc6d4488762b2e95"} Jul 13 14:53:07.699281 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.699253 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" event={"ID":"d886f0d5-ada3-4918-8536-03fd6a632ed3","Type":"ContainerStarted","Data":"13e0cc5249c0494578e0e109c64205c35c0e88e6726a0fbc6721320f43984425"} Jul 13 14:53:07.699564 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.699323 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" Jul 13 14:53:07.722315 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.722271 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" podStartSLOduration=1.271336862 podStartE2EDuration="4.722254858s" totalImagesPullingTime="3.450917996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:07.720896751 +0000 UTC m=+425.374863205" watchObservedRunningTime="2026-07-13 14:53:07.722254858 +0000 UTC m=+425.376221312" Jul 13 14:53:07.735137 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.735108 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm"] Jul 13 14:53:07.738501 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.738479 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" Jul 13 14:53:07.741254 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.741230 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dppnv\"" Jul 13 14:53:07.750341 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.750317 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm"] Jul 13 14:53:07.914178 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:07.914078 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvd4m\" (UniqueName: \"kubernetes.io/projected/60648541-e31b-4b1b-b803-95f4a92ab6ce-kube-api-access-cvd4m\") pod \"authorino-operator-75cc8d5cb7-7dgcm\" (UID: \"60648541-e31b-4b1b-b803-95f4a92ab6ce\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" Jul 13 14:53:08.015018 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:08.014986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvd4m\" (UniqueName: \"kubernetes.io/projected/60648541-e31b-4b1b-b803-95f4a92ab6ce-kube-api-access-cvd4m\") pod \"authorino-operator-75cc8d5cb7-7dgcm\" (UID: \"60648541-e31b-4b1b-b803-95f4a92ab6ce\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" Jul 13 14:53:08.029884 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:08.029855 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvd4m\" (UniqueName: \"kubernetes.io/projected/60648541-e31b-4b1b-b803-95f4a92ab6ce-kube-api-access-cvd4m\") pod \"authorino-operator-75cc8d5cb7-7dgcm\" (UID: \"60648541-e31b-4b1b-b803-95f4a92ab6ce\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" Jul 13 14:53:08.049807 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:08.049786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" Jul 13 14:53:08.181536 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:08.181408 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm"] Jul 13 14:53:08.184393 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:08.184355 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60648541_e31b_4b1b_b803_95f4a92ab6ce.slice/crio-aa3a5614751aff32e06ae4e02554d76bf095380106ca90d24338a0a0fc792152 WatchSource:0}: Error finding container aa3a5614751aff32e06ae4e02554d76bf095380106ca90d24338a0a0fc792152: Status 404 returned error can't find the container with id aa3a5614751aff32e06ae4e02554d76bf095380106ca90d24338a0a0fc792152 Jul 13 14:53:08.707174 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:08.707120 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" event={"ID":"60648541-e31b-4b1b-b803-95f4a92ab6ce","Type":"ContainerStarted","Data":"aa3a5614751aff32e06ae4e02554d76bf095380106ca90d24338a0a0fc792152"} Jul 13 14:53:10.718907 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:10.718864 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" event={"ID":"60648541-e31b-4b1b-b803-95f4a92ab6ce","Type":"ContainerStarted","Data":"3f6ef5da6ace83cc841a1093937dc73b7ca522be1bb33c142fd7b643344b189b"} Jul 13 14:53:10.719356 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:10.718984 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" Jul 13 14:53:10.738228 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:10.738172 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" podStartSLOduration=1.413091854 podStartE2EDuration="3.738145036s" totalImagesPullingTime="2.325053182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:10.73612587 +0000 UTC m=+428.390092335" watchObservedRunningTime="2026-07-13 14:53:10.738145036 +0000 UTC m=+428.392111490" Jul 13 14:53:11.115528 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.115494 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp"] Jul 13 14:53:11.119086 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.119070 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:11.123220 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.123201 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5mq4g\"" Jul 13 14:53:11.134338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.134309 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp"] Jul 13 14:53:11.244757 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.244726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttss\" (UniqueName: \"kubernetes.io/projected/dd8ab301-1bed-49fa-858a-56ea795ee32a-kube-api-access-9ttss\") pod \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" (UID: \"dd8ab301-1bed-49fa-858a-56ea795ee32a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:11.345828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.345796 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttss\" (UniqueName: \"kubernetes.io/projected/dd8ab301-1bed-49fa-858a-56ea795ee32a-kube-api-access-9ttss\") pod \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" (UID: \"dd8ab301-1bed-49fa-858a-56ea795ee32a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:11.354569 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.354539 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttss\" (UniqueName: \"kubernetes.io/projected/dd8ab301-1bed-49fa-858a-56ea795ee32a-kube-api-access-9ttss\") pod \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" (UID: \"dd8ab301-1bed-49fa-858a-56ea795ee32a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:11.429465 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.429364 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:11.564061 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.564026 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp"] Jul 13 14:53:11.566951 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:11.566919 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8ab301_1bed_49fa_858a_56ea795ee32a.slice/crio-e7518db0ddca27c56b28c838f7d0a90155fe1a5094997984cc7533babe703690 WatchSource:0}: Error finding container e7518db0ddca27c56b28c838f7d0a90155fe1a5094997984cc7533babe703690: Status 404 returned error can't find the container with id e7518db0ddca27c56b28c838f7d0a90155fe1a5094997984cc7533babe703690 Jul 13 14:53:11.725285 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:11.725196 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" event={"ID":"dd8ab301-1bed-49fa-858a-56ea795ee32a","Type":"ContainerStarted","Data":"e7518db0ddca27c56b28c838f7d0a90155fe1a5094997984cc7533babe703690"} Jul 13 14:53:12.864187 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:12.864131 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:12.866418 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:12.866389 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:12.872245 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:12.872223 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:13.738705 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:13.738608 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" event={"ID":"dd8ab301-1bed-49fa-858a-56ea795ee32a","Type":"ContainerStarted","Data":"1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3"} Jul 13 14:53:13.738864 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:13.738755 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:13.748197 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:13.748147 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d65f5fdf-qbqcw" Jul 13 14:53:13.758355 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:13.758308 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" podStartSLOduration=0.887644824 podStartE2EDuration="2.758294145s" totalImagesPullingTime="1.870649321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:13.756248545 +0000 UTC m=+431.410215000" watchObservedRunningTime="2026-07-13 14:53:13.758294145 +0000 UTC m=+431.412260599" Jul 13 14:53:13.812816 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:13.812785 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7c59d57-jftcv"] Jul 13 14:53:15.541144 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.541112 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7"] Jul 13 14:53:15.544821 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.544798 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.547631 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.547614 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qvsps\"" Jul 13 14:53:15.562189 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.562146 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7"] Jul 13 14:53:15.583440 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.583413 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77375f08-992f-4b59-bbcf-b84e38dab6e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.583601 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.583502 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl2f\" (UniqueName: \"kubernetes.io/projected/77375f08-992f-4b59-bbcf-b84e38dab6e7-kube-api-access-lsl2f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.683953 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.683921 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77375f08-992f-4b59-bbcf-b84e38dab6e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.684141 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.683984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl2f\" (UniqueName: \"kubernetes.io/projected/77375f08-992f-4b59-bbcf-b84e38dab6e7-kube-api-access-lsl2f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.684391 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.684368 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77375f08-992f-4b59-bbcf-b84e38dab6e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.696856 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.696824 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl2f\" (UniqueName: \"kubernetes.io/projected/77375f08-992f-4b59-bbcf-b84e38dab6e7-kube-api-access-lsl2f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.855055 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.854973 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:15.993436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:15.993390 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7"] Jul 13 14:53:15.995665 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:15.995640 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77375f08_992f_4b59_bbcf_b84e38dab6e7.slice/crio-bdf6b4b2ebae1915a443aebc58b95e7687486fba71cc96556051249dcbf94759 WatchSource:0}: Error finding container bdf6b4b2ebae1915a443aebc58b95e7687486fba71cc96556051249dcbf94759: Status 404 returned error can't find the container with id bdf6b4b2ebae1915a443aebc58b95e7687486fba71cc96556051249dcbf94759 Jul 13 14:53:16.754808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:16.754772 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" event={"ID":"77375f08-992f-4b59-bbcf-b84e38dab6e7","Type":"ContainerStarted","Data":"bdf6b4b2ebae1915a443aebc58b95e7687486fba71cc96556051249dcbf94759"} Jul 13 14:53:18.709607 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:18.709579 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-c9lkx" Jul 13 14:53:21.728176 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:21.728121 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dgcm" Jul 13 14:53:22.787018 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:22.786984 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" event={"ID":"77375f08-992f-4b59-bbcf-b84e38dab6e7","Type":"ContainerStarted","Data":"294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd"} Jul 13 14:53:22.787460 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:22.787125 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:22.824944 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:22.824898 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" podStartSLOduration=1.8864020099999999 podStartE2EDuration="7.824882002s" totalImagesPullingTime="5.938479992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:22.821439342 +0000 UTC m=+440.475405797" watchObservedRunningTime="2026-07-13 14:53:22.824882002 +0000 UTC m=+440.478848455" Jul 13 14:53:24.600624 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.600568 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl"] Jul 13 14:53:24.604165 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.604138 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.607141 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.607124 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 13 14:53:24.607515 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.607491 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v8c7p\"" Jul 13 14:53:24.607782 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.607756 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 13 14:53:24.613039 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.613018 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl"] Jul 13 14:53:24.671657 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.671627 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/841cdb67-b6ff-4426-b2ed-b149c45a4ace-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.671788 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.671684 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjb7\" (UniqueName: \"kubernetes.io/projected/841cdb67-b6ff-4426-b2ed-b149c45a4ace-kube-api-access-vdjb7\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.671788 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.671751 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/841cdb67-b6ff-4426-b2ed-b149c45a4ace-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.746488 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.746461 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:24.772336 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.772305 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/841cdb67-b6ff-4426-b2ed-b149c45a4ace-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.772554 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.772391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/841cdb67-b6ff-4426-b2ed-b149c45a4ace-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.772554 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.772448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjb7\" (UniqueName: \"kubernetes.io/projected/841cdb67-b6ff-4426-b2ed-b149c45a4ace-kube-api-access-vdjb7\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.773377 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.773352 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/841cdb67-b6ff-4426-b2ed-b149c45a4ace-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.775344 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.775319 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/841cdb67-b6ff-4426-b2ed-b149c45a4ace-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.781968 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.781940 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjb7\" (UniqueName: \"kubernetes.io/projected/841cdb67-b6ff-4426-b2ed-b149c45a4ace-kube-api-access-vdjb7\") pod \"kuadrant-console-plugin-66b5774985-nj4cl\" (UID: \"841cdb67-b6ff-4426-b2ed-b149c45a4ace\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:24.915038 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:24.914965 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" Jul 13 14:53:25.052362 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:25.052338 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl"] Jul 13 14:53:25.054574 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:25.054543 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod841cdb67_b6ff_4426_b2ed_b149c45a4ace.slice/crio-e5d9cf38ebc2d7bc1497b4070c2413b986f24ce4bf1ca9941d00b931b7ee7ffe WatchSource:0}: Error finding container e5d9cf38ebc2d7bc1497b4070c2413b986f24ce4bf1ca9941d00b931b7ee7ffe: Status 404 returned error can't find the container with id e5d9cf38ebc2d7bc1497b4070c2413b986f24ce4bf1ca9941d00b931b7ee7ffe Jul 13 14:53:25.802341 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:25.802302 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" event={"ID":"841cdb67-b6ff-4426-b2ed-b149c45a4ace","Type":"ContainerStarted","Data":"e5d9cf38ebc2d7bc1497b4070c2413b986f24ce4bf1ca9941d00b931b7ee7ffe"} Jul 13 14:53:33.794979 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:33.794943 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:34.802750 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:34.802712 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq"] Jul 13 14:53:34.811118 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:34.811091 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:34.819698 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:34.819654 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq"] Jul 13 14:53:34.970138 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:34.970095 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:34.970350 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:34.970192 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhvt5\" (UniqueName: \"kubernetes.io/projected/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-kube-api-access-lhvt5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:35.071639 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.071554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:35.071639 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.071624 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvt5\" (UniqueName: \"kubernetes.io/projected/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-kube-api-access-lhvt5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:35.071960 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.071938 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:35.089163 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.089128 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhvt5\" (UniqueName: \"kubernetes.io/projected/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-kube-api-access-lhvt5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:35.125087 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.125034 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:35.615264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.615231 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq"] Jul 13 14:53:35.627145 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.627112 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq"] Jul 13 14:53:35.639189 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.639121 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7"] Jul 13 14:53:35.639712 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.639367 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" containerName="manager" containerID="cri-o://294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd" gracePeriod=2 Jul 13 14:53:35.642611 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.642584 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7"] Jul 13 14:53:35.643398 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:35.643351 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:35.651181 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.651124 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v"] Jul 13 14:53:35.651973 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.651701 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" containerName="manager" Jul 13 14:53:35.651973 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.651725 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" containerName="manager" Jul 13 14:53:35.651973 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.651827 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" containerName="manager" Jul 13 14:53:35.659293 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.659272 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp"] Jul 13 14:53:35.659903 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.659380 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:35.659903 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.659568 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" containerName="manager" containerID="cri-o://1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3" gracePeriod=2 Jul 13 14:53:35.664422 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:35.663365 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:35.667273 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.666958 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp"] Jul 13 14:53:35.667273 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:35.667116 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:35.689068 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.685187 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v"] Jul 13 14:53:35.689438 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.689410 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv"] Jul 13 14:53:35.693180 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.693120 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" containerName="manager" Jul 13 14:53:35.693180 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.693146 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" containerName="manager" Jul 13 14:53:35.693547 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.693313 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" containerName="manager" Jul 13 14:53:35.697818 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:35.697781 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:35.698736 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.698281 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:35.701645 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:35.701613 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:35.702576 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.702549 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv"] Jul 13 14:53:35.706986 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:35.706955 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:35.735333 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:35.735287 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:35.779053 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.779017 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwmcv\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:35.779251 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.779134 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37693ca0-9f31-49de-9592-a5af88cfdc70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8st5v\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:35.779319 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.779247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdzw\" (UniqueName: \"kubernetes.io/projected/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-kube-api-access-qmdzw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwmcv\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:35.779319 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.779276 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmzv\" (UniqueName: \"kubernetes.io/projected/37693ca0-9f31-49de-9592-a5af88cfdc70-kube-api-access-kvmzv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8st5v\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:35.880811 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.880728 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdzw\" (UniqueName: \"kubernetes.io/projected/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-kube-api-access-qmdzw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwmcv\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:35.880811 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.880774 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmzv\" (UniqueName: \"kubernetes.io/projected/37693ca0-9f31-49de-9592-a5af88cfdc70-kube-api-access-kvmzv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8st5v\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:35.881280 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.880822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwmcv\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:35.881280 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.880903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37693ca0-9f31-49de-9592-a5af88cfdc70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8st5v\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:35.881280 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.881273 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwmcv\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:35.881406 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.881284 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37693ca0-9f31-49de-9592-a5af88cfdc70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8st5v\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:35.890788 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.890745 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdzw\" (UniqueName: \"kubernetes.io/projected/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-kube-api-access-qmdzw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwmcv\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:35.890927 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:35.890877 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmzv\" (UniqueName: \"kubernetes.io/projected/37693ca0-9f31-49de-9592-a5af88cfdc70-kube-api-access-kvmzv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8st5v\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:36.042461 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:36.042423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:36.050555 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:36.050520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:38.836513 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:38.836475 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67f7c59d57-jftcv" podUID="e326671c-df0a-4235-aecd-69e3cc8579c4" containerName="console" containerID="cri-o://21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8" gracePeriod=15 Jul 13 14:53:45.725577 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:45.725534 2571 patch_prober.go:28] interesting pod/console-67f7c59d57-jftcv container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" start-of-body= Jul 13 14:53:45.725966 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:45.725603 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-67f7c59d57-jftcv" podUID="e326671c-df0a-4235-aecd-69e3cc8579c4" containerName="console" probeResult="failure" output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" Jul 13 14:53:48.348051 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.348020 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:53:48.351070 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.351002 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:48.353215 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.353195 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv"] Jul 13 14:53:48.356059 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.356022 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:48.357141 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:48.357097 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd0fb21_ee7b_4e7e_aaa5_02d36e425b0e.slice/crio-8fa6363f3a3461d3e6e81b071948c43bf00f6a45fa1252c2486ce349087b227e WatchSource:0}: Error finding container 8fa6363f3a3461d3e6e81b071948c43bf00f6a45fa1252c2486ce349087b227e: Status 404 returned error can't find the container with id 8fa6363f3a3461d3e6e81b071948c43bf00f6a45fa1252c2486ce349087b227e Jul 13 14:53:48.369255 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:48.369223 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:48.373482 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:48.373444 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:48.375845 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.375781 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v"] Jul 13 14:53:48.378255 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:53:48.378222 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37693ca0_9f31_49de_9592_a5af88cfdc70.slice/crio-c978e6e02f32beb6d10a651b7f5b85b6900e5d9c668ede34c961be744d4a5051 WatchSource:0}: Error finding container c978e6e02f32beb6d10a651b7f5b85b6900e5d9c668ede34c961be744d4a5051: Status 404 returned error can't find the container with id c978e6e02f32beb6d10a651b7f5b85b6900e5d9c668ede34c961be744d4a5051 Jul 13 14:53:48.413264 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413227 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-oauth-config\") pod \"e326671c-df0a-4235-aecd-69e3cc8579c4\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " Jul 13 14:53:48.413380 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413275 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-serving-cert\") pod \"e326671c-df0a-4235-aecd-69e3cc8579c4\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " Jul 13 14:53:48.413380 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413308 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsl2f\" (UniqueName: \"kubernetes.io/projected/77375f08-992f-4b59-bbcf-b84e38dab6e7-kube-api-access-lsl2f\") pod \"77375f08-992f-4b59-bbcf-b84e38dab6e7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " Jul 13 14:53:48.413380 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413333 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgdbn\" (UniqueName: \"kubernetes.io/projected/e326671c-df0a-4235-aecd-69e3cc8579c4-kube-api-access-pgdbn\") pod \"e326671c-df0a-4235-aecd-69e3cc8579c4\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " Jul 13 14:53:48.413537 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413388 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-trusted-ca-bundle\") pod \"e326671c-df0a-4235-aecd-69e3cc8579c4\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " Jul 13 14:53:48.413537 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413425 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-oauth-serving-cert\") pod \"e326671c-df0a-4235-aecd-69e3cc8579c4\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " Jul 13 14:53:48.413537 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413464 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77375f08-992f-4b59-bbcf-b84e38dab6e7-extensions-socket-volume\") pod \"77375f08-992f-4b59-bbcf-b84e38dab6e7\" (UID: \"77375f08-992f-4b59-bbcf-b84e38dab6e7\") " Jul 13 14:53:48.413537 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413495 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-service-ca\") pod \"e326671c-df0a-4235-aecd-69e3cc8579c4\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " Jul 13 14:53:48.413537 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413524 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ttss\" (UniqueName: \"kubernetes.io/projected/dd8ab301-1bed-49fa-858a-56ea795ee32a-kube-api-access-9ttss\") pod \"dd8ab301-1bed-49fa-858a-56ea795ee32a\" (UID: \"dd8ab301-1bed-49fa-858a-56ea795ee32a\") " Jul 13 14:53:48.413769 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413591 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-console-config\") pod \"e326671c-df0a-4235-aecd-69e3cc8579c4\" (UID: \"e326671c-df0a-4235-aecd-69e3cc8579c4\") " Jul 13 14:53:48.413897 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413867 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e326671c-df0a-4235-aecd-69e3cc8579c4" (UID: "e326671c-df0a-4235-aecd-69e3cc8579c4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:48.414242 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.413977 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e326671c-df0a-4235-aecd-69e3cc8579c4" (UID: "e326671c-df0a-4235-aecd-69e3cc8579c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:48.414242 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.414124 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77375f08-992f-4b59-bbcf-b84e38dab6e7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "77375f08-992f-4b59-bbcf-b84e38dab6e7" (UID: "77375f08-992f-4b59-bbcf-b84e38dab6e7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:48.414419 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.414330 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-service-ca" (OuterVolumeSpecName: "service-ca") pod "e326671c-df0a-4235-aecd-69e3cc8579c4" (UID: "e326671c-df0a-4235-aecd-69e3cc8579c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:48.414619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.414585 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-console-config" (OuterVolumeSpecName: "console-config") pod "e326671c-df0a-4235-aecd-69e3cc8579c4" (UID: "e326671c-df0a-4235-aecd-69e3cc8579c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:48.415564 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.415535 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e326671c-df0a-4235-aecd-69e3cc8579c4" (UID: "e326671c-df0a-4235-aecd-69e3cc8579c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:53:48.416146 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.416116 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e326671c-df0a-4235-aecd-69e3cc8579c4" (UID: "e326671c-df0a-4235-aecd-69e3cc8579c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:53:48.416526 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.416497 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e326671c-df0a-4235-aecd-69e3cc8579c4-kube-api-access-pgdbn" (OuterVolumeSpecName: "kube-api-access-pgdbn") pod "e326671c-df0a-4235-aecd-69e3cc8579c4" (UID: "e326671c-df0a-4235-aecd-69e3cc8579c4"). InnerVolumeSpecName "kube-api-access-pgdbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:48.416526 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.416514 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8ab301-1bed-49fa-858a-56ea795ee32a-kube-api-access-9ttss" (OuterVolumeSpecName: "kube-api-access-9ttss") pod "dd8ab301-1bed-49fa-858a-56ea795ee32a" (UID: "dd8ab301-1bed-49fa-858a-56ea795ee32a"). InnerVolumeSpecName "kube-api-access-9ttss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:48.416641 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.416594 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77375f08-992f-4b59-bbcf-b84e38dab6e7-kube-api-access-lsl2f" (OuterVolumeSpecName: "kube-api-access-lsl2f") pod "77375f08-992f-4b59-bbcf-b84e38dab6e7" (UID: "77375f08-992f-4b59-bbcf-b84e38dab6e7"). InnerVolumeSpecName "kube-api-access-lsl2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:48.514470 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514421 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-serving-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514470 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514463 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsl2f\" (UniqueName: \"kubernetes.io/projected/77375f08-992f-4b59-bbcf-b84e38dab6e7-kube-api-access-lsl2f\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514470 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514474 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgdbn\" (UniqueName: \"kubernetes.io/projected/e326671c-df0a-4235-aecd-69e3cc8579c4-kube-api-access-pgdbn\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514470 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514482 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-trusted-ca-bundle\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514470 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514491 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-oauth-serving-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514769 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514499 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77375f08-992f-4b59-bbcf-b84e38dab6e7-extensions-socket-volume\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514769 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514508 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-service-ca\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514769 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514517 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ttss\" (UniqueName: \"kubernetes.io/projected/dd8ab301-1bed-49fa-858a-56ea795ee32a-kube-api-access-9ttss\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514769 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514525 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e326671c-df0a-4235-aecd-69e3cc8579c4-console-config\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.514769 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.514535 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e326671c-df0a-4235-aecd-69e3cc8579c4-console-oauth-config\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:48.870961 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.870924 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" path="/var/lib/kubelet/pods/77375f08-992f-4b59-bbcf-b84e38dab6e7/volumes" Jul 13 14:53:48.871332 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.871316 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" path="/var/lib/kubelet/pods/dd8ab301-1bed-49fa-858a-56ea795ee32a/volumes" Jul 13 14:53:48.931613 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.931583 2571 generic.go:356] "Generic (PLEG): container finished" podID="e326671c-df0a-4235-aecd-69e3cc8579c4" containerID="21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8" exitCode=0 Jul 13 14:53:48.931738 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.931637 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c59d57-jftcv" Jul 13 14:53:48.931738 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.931665 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c59d57-jftcv" event={"ID":"e326671c-df0a-4235-aecd-69e3cc8579c4","Type":"ContainerDied","Data":"21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8"} Jul 13 14:53:48.931738 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.931715 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c59d57-jftcv" event={"ID":"e326671c-df0a-4235-aecd-69e3cc8579c4","Type":"ContainerDied","Data":"c1944e2c7de2f1f5032351ffeaae389ac34e35efdd6696b0c7e076b1470e5405"} Jul 13 14:53:48.931915 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.931740 2571 scope.go:117] "RemoveContainer" containerID="21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8" Jul 13 14:53:48.933081 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.933060 2571 generic.go:356] "Generic (PLEG): container finished" podID="77375f08-992f-4b59-bbcf-b84e38dab6e7" containerID="294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd" exitCode=0 Jul 13 14:53:48.933204 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.933108 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:48.939056 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.938472 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" event={"ID":"841cdb67-b6ff-4426-b2ed-b149c45a4ace","Type":"ContainerStarted","Data":"f2e5f4662c9391cc7f08f72cc9e071ebe3524245ff15a786354b3e6a68d41ca5"} Jul 13 14:53:48.942726 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.942698 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" event={"ID":"37693ca0-9f31-49de-9592-a5af88cfdc70","Type":"ContainerStarted","Data":"d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264"} Jul 13 14:53:48.942842 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.942745 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" event={"ID":"37693ca0-9f31-49de-9592-a5af88cfdc70","Type":"ContainerStarted","Data":"c978e6e02f32beb6d10a651b7f5b85b6900e5d9c668ede34c961be744d4a5051"} Jul 13 14:53:48.942842 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.942772 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:48.943995 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.943975 2571 generic.go:356] "Generic (PLEG): container finished" podID="dd8ab301-1bed-49fa-858a-56ea795ee32a" containerID="1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3" exitCode=0 Jul 13 14:53:48.944111 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.944009 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:48.946897 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.946879 2571 scope.go:117] "RemoveContainer" containerID="21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8" Jul 13 14:53:48.947715 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:48.947686 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:48.947826 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:48.947694 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8\": container with ID starting with 21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8 not found: ID does not exist" containerID="21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8" Jul 13 14:53:48.947826 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.947787 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8"} err="failed to get container status \"21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8\": rpc error: code = NotFound desc = could not find container \"21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8\": container with ID starting with 21ddcf4cad1676fd9ec1f4b75725706bd69c570dca4805cd21dc6be687426fb8 not found: ID does not exist" Jul 13 14:53:48.947826 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.947813 2571 scope.go:117] "RemoveContainer" containerID="294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd" Jul 13 14:53:48.949064 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.949025 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" event={"ID":"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e","Type":"ContainerStarted","Data":"490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310"} Jul 13 14:53:48.949131 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.949070 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" event={"ID":"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e","Type":"ContainerStarted","Data":"8fa6363f3a3461d3e6e81b071948c43bf00f6a45fa1252c2486ce349087b227e"} Jul 13 14:53:48.949131 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.949114 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:53:48.956341 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.956325 2571 scope.go:117] "RemoveContainer" containerID="294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd" Jul 13 14:53:48.956580 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:48.956559 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd\": container with ID starting with 294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd not found: ID does not exist" containerID="294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd" Jul 13 14:53:48.956625 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.956587 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd"} err="failed to get container status \"294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd\": rpc error: code = NotFound desc = could not find container \"294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd\": container with ID starting with 294c834fdf9ef666d08a6f6812bb5fb1ff8c72b9ce3297cfebc899533af4e4dd not found: ID does not exist" Jul 13 14:53:48.956625 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.956602 2571 scope.go:117] "RemoveContainer" containerID="1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3" Jul 13 14:53:48.963904 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.963887 2571 scope.go:117] "RemoveContainer" containerID="1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3" Jul 13 14:53:48.964121 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:48.964101 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3\": container with ID starting with 1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3 not found: ID does not exist" containerID="1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3" Jul 13 14:53:48.964196 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.964126 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3"} err="failed to get container status \"1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3\": rpc error: code = NotFound desc = could not find container \"1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3\": container with ID starting with 1b3b90cb73bd383116dc8c5cbd7f7af47fac5acf2683775a023b4cd322345ed3 not found: ID does not exist" Jul 13 14:53:48.972998 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.972979 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7c59d57-jftcv"] Jul 13 14:53:48.979244 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.979227 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f7c59d57-jftcv"] Jul 13 14:53:48.998096 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:48.998059 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nj4cl" podStartSLOduration=1.798084454 podStartE2EDuration="24.998048402s" totalImagesPullingTime="23.199963948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:48.997115016 +0000 UTC m=+466.651081471" watchObservedRunningTime="2026-07-13 14:53:48.998048402 +0000 UTC m=+466.652014855" Jul 13 14:53:48.999670 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:48.999649 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6nph7\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="77375f08-992f-4b59-bbcf-b84e38dab6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6nph7" Jul 13 14:53:49.024789 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:49.024744 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" podStartSLOduration=14.024728968 podStartE2EDuration="14.024728968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:49.023197342 +0000 UTC m=+466.677163796" watchObservedRunningTime="2026-07-13 14:53:49.024728968 +0000 UTC m=+466.678695421" Jul 13 14:53:49.025555 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:49.025522 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wr5gp\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="dd8ab301-1bed-49fa-858a-56ea795ee32a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wr5gp" Jul 13 14:53:49.051343 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:49.051298 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" podStartSLOduration=14.051283304 podStartE2EDuration="14.051283304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:49.04971482 +0000 UTC m=+466.703681274" watchObservedRunningTime="2026-07-13 14:53:49.051283304 +0000 UTC m=+466.705249761" Jul 13 14:53:50.702383 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:50.702345 2571 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-7pwtq_kuadrant-system_08e0ed96-fcae-43e7-bc52-5cfed28df6a9_0(df7452fb3f535a4a87ecc13fba6fa124bff0e6d7263468fc5a84a69c194b1f0b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-7pwtq to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 13 14:53:50.702648 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:50.702409 2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-7pwtq_kuadrant-system_08e0ed96-fcae-43e7-bc52-5cfed28df6a9_0(df7452fb3f535a4a87ecc13fba6fa124bff0e6d7263468fc5a84a69c194b1f0b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-7pwtq to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:50.871210 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:50.871129 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e326671c-df0a-4235-aecd-69e3cc8579c4" path="/var/lib/kubelet/pods/e326671c-df0a-4235-aecd-69e3cc8579c4/volumes" Jul 13 14:53:50.960947 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:50.960872 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:50.963572 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:50.963541 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="08e0ed96-fcae-43e7-bc52-5cfed28df6a9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:50.970629 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:50.970607 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:50.972993 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:50.972970 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="08e0ed96-fcae-43e7-bc52-5cfed28df6a9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:51.036557 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:51.036527 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-extensions-socket-volume\") pod \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " Jul 13 14:53:51.036653 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:51.036566 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhvt5\" (UniqueName: \"kubernetes.io/projected/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-kube-api-access-lhvt5\") pod \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\" (UID: \"08e0ed96-fcae-43e7-bc52-5cfed28df6a9\") " Jul 13 14:53:51.036814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:51.036792 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "08e0ed96-fcae-43e7-bc52-5cfed28df6a9" (UID: "08e0ed96-fcae-43e7-bc52-5cfed28df6a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:51.038645 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:51.038620 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-kube-api-access-lhvt5" (OuterVolumeSpecName: "kube-api-access-lhvt5") pod "08e0ed96-fcae-43e7-bc52-5cfed28df6a9" (UID: "08e0ed96-fcae-43e7-bc52-5cfed28df6a9"). InnerVolumeSpecName "kube-api-access-lhvt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:51.137611 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:51.137580 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-extensions-socket-volume\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:51.137611 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:51.137607 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhvt5\" (UniqueName: \"kubernetes.io/projected/08e0ed96-fcae-43e7-bc52-5cfed28df6a9-kube-api-access-lhvt5\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:53:51.965894 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:51.965864 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:51.968173 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:51.968132 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="08e0ed96-fcae-43e7-bc52-5cfed28df6a9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:51.973610 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:51.973579 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="08e0ed96-fcae-43e7-bc52-5cfed28df6a9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:52.869960 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:53:52.869920 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7pwtq\" is forbidden: User \"system:node:ip-10-0-133-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-41.ec2.internal' and this object" podUID="08e0ed96-fcae-43e7-bc52-5cfed28df6a9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7pwtq" Jul 13 14:53:52.872669 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:52.872646 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e0ed96-fcae-43e7-bc52-5cfed28df6a9" path="/var/lib/kubelet/pods/08e0ed96-fcae-43e7-bc52-5cfed28df6a9/volumes" Jul 13 14:53:59.959442 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:59.959410 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:53:59.959928 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:53:59.959483 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:54:00.040416 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.040377 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv"] Jul 13 14:54:00.040655 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.040633 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" podUID="5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" containerName="manager" containerID="cri-o://490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310" gracePeriod=10 Jul 13 14:54:00.259214 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.259114 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l"] Jul 13 14:54:00.259535 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.259521 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e326671c-df0a-4235-aecd-69e3cc8579c4" containerName="console" Jul 13 14:54:00.259623 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.259537 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e326671c-df0a-4235-aecd-69e3cc8579c4" containerName="console" Jul 13 14:54:00.259623 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.259604 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e326671c-df0a-4235-aecd-69e3cc8579c4" containerName="console" Jul 13 14:54:00.262967 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.262945 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.278368 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.278335 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l"] Jul 13 14:54:00.297652 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.297628 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:54:00.318665 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.318628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d4459c-e569-4ce4-aa24-2335440533da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tnv9l\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.318806 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.318686 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnqc\" (UniqueName: \"kubernetes.io/projected/85d4459c-e569-4ce4-aa24-2335440533da-kube-api-access-9rnqc\") pod \"kuadrant-operator-controller-manager-86878c769f-tnv9l\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.420051 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.420023 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-extensions-socket-volume\") pod \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " Jul 13 14:54:00.420257 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.420069 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmdzw\" (UniqueName: \"kubernetes.io/projected/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-kube-api-access-qmdzw\") pod \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\" (UID: \"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e\") " Jul 13 14:54:00.420257 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.420240 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnqc\" (UniqueName: \"kubernetes.io/projected/85d4459c-e569-4ce4-aa24-2335440533da-kube-api-access-9rnqc\") pod \"kuadrant-operator-controller-manager-86878c769f-tnv9l\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.420382 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.420364 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d4459c-e569-4ce4-aa24-2335440533da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tnv9l\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.420444 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.420425 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" (UID: "5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:54:00.420725 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.420704 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d4459c-e569-4ce4-aa24-2335440533da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tnv9l\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.422072 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.422048 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-kube-api-access-qmdzw" (OuterVolumeSpecName: "kube-api-access-qmdzw") pod "5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" (UID: "5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e"). InnerVolumeSpecName "kube-api-access-qmdzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:00.437094 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.437066 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnqc\" (UniqueName: \"kubernetes.io/projected/85d4459c-e569-4ce4-aa24-2335440533da-kube-api-access-9rnqc\") pod \"kuadrant-operator-controller-manager-86878c769f-tnv9l\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.521580 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.521494 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-extensions-socket-volume\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:54:00.521580 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.521524 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmdzw\" (UniqueName: \"kubernetes.io/projected/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e-kube-api-access-qmdzw\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:54:00.592812 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.592774 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:00.728371 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:00.728347 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l"] Jul 13 14:54:00.730284 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:54:00.730252 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d4459c_e569_4ce4_aa24_2335440533da.slice/crio-c4f4a04baa37a6e7d530e62442e31e8747896334b0f94e322c4a2914b590f857 WatchSource:0}: Error finding container c4f4a04baa37a6e7d530e62442e31e8747896334b0f94e322c4a2914b590f857: Status 404 returned error can't find the container with id c4f4a04baa37a6e7d530e62442e31e8747896334b0f94e322c4a2914b590f857 Jul 13 14:54:01.014055 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.014020 2571 generic.go:356] "Generic (PLEG): container finished" podID="5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" containerID="490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310" exitCode=0 Jul 13 14:54:01.014572 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.014080 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" Jul 13 14:54:01.014572 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.014105 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" event={"ID":"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e","Type":"ContainerDied","Data":"490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310"} Jul 13 14:54:01.014572 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.014172 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv" event={"ID":"5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e","Type":"ContainerDied","Data":"8fa6363f3a3461d3e6e81b071948c43bf00f6a45fa1252c2486ce349087b227e"} Jul 13 14:54:01.014572 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.014192 2571 scope.go:117] "RemoveContainer" containerID="490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310" Jul 13 14:54:01.016025 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.015995 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" event={"ID":"85d4459c-e569-4ce4-aa24-2335440533da","Type":"ContainerStarted","Data":"346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd"} Jul 13 14:54:01.016169 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.016049 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" event={"ID":"85d4459c-e569-4ce4-aa24-2335440533da","Type":"ContainerStarted","Data":"c4f4a04baa37a6e7d530e62442e31e8747896334b0f94e322c4a2914b590f857"} Jul 13 14:54:01.016169 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.016142 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:01.022828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.022813 2571 scope.go:117] "RemoveContainer" containerID="490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310" Jul 13 14:54:01.023071 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:54:01.023052 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310\": container with ID starting with 490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310 not found: ID does not exist" containerID="490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310" Jul 13 14:54:01.023146 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.023082 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310"} err="failed to get container status \"490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310\": rpc error: code = NotFound desc = could not find container \"490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310\": container with ID starting with 490f24db764225c1943af435cab3cfd0f562c4fe24c8f441f01366756d5f5310 not found: ID does not exist" Jul 13 14:54:01.036776 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.036749 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv"] Jul 13 14:54:01.040840 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.040816 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwmcv"] Jul 13 14:54:01.058747 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:01.058535 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" podStartSLOduration=1.058510111 podStartE2EDuration="1.058510111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:01.058180661 +0000 UTC m=+478.712147112" watchObservedRunningTime="2026-07-13 14:54:01.058510111 +0000 UTC m=+478.712476571" Jul 13 14:54:02.875833 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:02.875797 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" path="/var/lib/kubelet/pods/5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e/volumes" Jul 13 14:54:12.025435 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:12.025403 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 14:54:12.080940 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:12.080909 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v"] Jul 13 14:54:12.081126 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:12.081104 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" podUID="37693ca0-9f31-49de-9592-a5af88cfdc70" containerName="manager" containerID="cri-o://d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264" gracePeriod=10 Jul 13 14:54:12.883663 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:12.883633 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:54:13.034140 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.034114 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37693ca0-9f31-49de-9592-a5af88cfdc70-extensions-socket-volume\") pod \"37693ca0-9f31-49de-9592-a5af88cfdc70\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " Jul 13 14:54:13.034140 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.034148 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvmzv\" (UniqueName: \"kubernetes.io/projected/37693ca0-9f31-49de-9592-a5af88cfdc70-kube-api-access-kvmzv\") pod \"37693ca0-9f31-49de-9592-a5af88cfdc70\" (UID: \"37693ca0-9f31-49de-9592-a5af88cfdc70\") " Jul 13 14:54:13.034685 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.034556 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37693ca0-9f31-49de-9592-a5af88cfdc70-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37693ca0-9f31-49de-9592-a5af88cfdc70" (UID: "37693ca0-9f31-49de-9592-a5af88cfdc70"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:54:13.036387 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.036361 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37693ca0-9f31-49de-9592-a5af88cfdc70-kube-api-access-kvmzv" (OuterVolumeSpecName: "kube-api-access-kvmzv") pod "37693ca0-9f31-49de-9592-a5af88cfdc70" (UID: "37693ca0-9f31-49de-9592-a5af88cfdc70"). InnerVolumeSpecName "kube-api-access-kvmzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:13.078620 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.078589 2571 generic.go:356] "Generic (PLEG): container finished" podID="37693ca0-9f31-49de-9592-a5af88cfdc70" containerID="d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264" exitCode=0 Jul 13 14:54:13.078752 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.078642 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" Jul 13 14:54:13.078752 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.078658 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" event={"ID":"37693ca0-9f31-49de-9592-a5af88cfdc70","Type":"ContainerDied","Data":"d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264"} Jul 13 14:54:13.078752 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.078723 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v" event={"ID":"37693ca0-9f31-49de-9592-a5af88cfdc70","Type":"ContainerDied","Data":"c978e6e02f32beb6d10a651b7f5b85b6900e5d9c668ede34c961be744d4a5051"} Jul 13 14:54:13.078752 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.078752 2571 scope.go:117] "RemoveContainer" containerID="d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264" Jul 13 14:54:13.089053 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.088856 2571 scope.go:117] "RemoveContainer" containerID="d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264" Jul 13 14:54:13.089130 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:54:13.089112 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264\": container with ID starting with d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264 not found: ID does not exist" containerID="d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264" Jul 13 14:54:13.089288 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.089138 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264"} err="failed to get container status \"d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264\": rpc error: code = NotFound desc = could not find container \"d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264\": container with ID starting with d778e1ec8366b58875a8b05f6ac77921daa6495726890d377d8b4a278570f264 not found: ID does not exist" Jul 13 14:54:13.103847 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.103822 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v"] Jul 13 14:54:13.109746 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.109722 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8st5v"] Jul 13 14:54:13.135866 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.135841 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37693ca0-9f31-49de-9592-a5af88cfdc70-extensions-socket-volume\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:54:13.135866 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:13.135865 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvmzv\" (UniqueName: \"kubernetes.io/projected/37693ca0-9f31-49de-9592-a5af88cfdc70-kube-api-access-kvmzv\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:54:14.875357 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:14.875323 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37693ca0-9f31-49de-9592-a5af88cfdc70" path="/var/lib/kubelet/pods/37693ca0-9f31-49de-9592-a5af88cfdc70/volumes" Jul 13 14:54:16.634276 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.634240 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx"] Jul 13 14:54:16.634770 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.634671 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" containerName="manager" Jul 13 14:54:16.634770 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.634689 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" containerName="manager" Jul 13 14:54:16.634770 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.634723 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37693ca0-9f31-49de-9592-a5af88cfdc70" containerName="manager" Jul 13 14:54:16.634770 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.634731 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="37693ca0-9f31-49de-9592-a5af88cfdc70" containerName="manager" Jul 13 14:54:16.634924 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.634830 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="37693ca0-9f31-49de-9592-a5af88cfdc70" containerName="manager" Jul 13 14:54:16.634924 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.634842 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd0fb21-ee7b-4e7e-aaa5-02d36e425b0e" containerName="manager" Jul 13 14:54:16.772833 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.772798 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx"] Jul 13 14:54:16.773015 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.772909 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.775397 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.775369 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tc4xs\"" Jul 13 14:54:16.775534 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.775469 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 14:54:16.869633 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869600 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869633 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869637 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52sq\" (UniqueName: \"kubernetes.io/projected/9280c649-ece0-4e47-be43-cca2b81c54b1-kube-api-access-l52sq\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869659 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869784 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869828 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9280c649-ece0-4e47-be43-cca2b81c54b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869960 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869860 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869960 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869880 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.869960 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.869898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970633 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970560 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970633 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970604 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970639 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9280c649-ece0-4e47-be43-cca2b81c54b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970724 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970750 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970798 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.970867 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970849 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52sq\" (UniqueName: \"kubernetes.io/projected/9280c649-ece0-4e47-be43-cca2b81c54b1-kube-api-access-l52sq\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.971240 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.970890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.971240 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.971166 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.971333 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.971234 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.971439 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.971422 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.971490 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.971432 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9280c649-ece0-4e47-be43-cca2b81c54b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.971529 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.971516 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.973206 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.973184 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.973462 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.973446 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.979548 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.979524 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9280c649-ece0-4e47-be43-cca2b81c54b1-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:16.979776 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:16.979757 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52sq\" (UniqueName: \"kubernetes.io/projected/9280c649-ece0-4e47-be43-cca2b81c54b1-kube-api-access-l52sq\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sjhsx\" (UID: \"9280c649-ece0-4e47-be43-cca2b81c54b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:17.084321 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:17.084289 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:17.269436 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:17.269357 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx"] Jul 13 14:54:17.272014 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:54:17.271991 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9280c649_ece0_4e47_be43_cca2b81c54b1.slice/crio-664b160f28495ec909dd7995bb5ecafc79e3b6a2940cefc23444676cf9613993 WatchSource:0}: Error finding container 664b160f28495ec909dd7995bb5ecafc79e3b6a2940cefc23444676cf9613993: Status 404 returned error can't find the container with id 664b160f28495ec909dd7995bb5ecafc79e3b6a2940cefc23444676cf9613993 Jul 13 14:54:18.102750 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:18.102712 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" event={"ID":"9280c649-ece0-4e47-be43-cca2b81c54b1","Type":"ContainerStarted","Data":"664b160f28495ec909dd7995bb5ecafc79e3b6a2940cefc23444676cf9613993"} Jul 13 14:54:20.251310 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:20.251269 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:54:20.251613 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:20.251341 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:54:20.251613 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:20.251370 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:54:21.120664 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:21.120628 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" event={"ID":"9280c649-ece0-4e47-be43-cca2b81c54b1","Type":"ContainerStarted","Data":"26d1bd67d9a1a2ef867939e12568ce3ec82ca30c5e555fa0b493aedddccb8332"} Jul 13 14:54:21.146598 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:21.146547 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" podStartSLOduration=2.169905828 podStartE2EDuration="5.146531955s" totalImagesPullingTime="2.976626127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:21.145465323 +0000 UTC m=+498.799431789" watchObservedRunningTime="2026-07-13 14:54:21.146531955 +0000 UTC m=+498.800498482" Jul 13 14:54:22.084639 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:22.084601 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:22.089465 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:22.089440 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:22.131391 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:22.131350 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:22.132255 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:22.132238 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:54:29.398707 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.398662 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cdssm"] Jul 13 14:54:29.402287 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.402262 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.404894 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.404873 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 13 14:54:29.411058 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.411033 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cdssm"] Jul 13 14:54:29.484751 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.484714 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c66cf258-b98d-4b90-baa5-e557462560dd-config-file\") pod \"limitador-limitador-64c94fd64f-cdssm\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.484986 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.484830 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftbk\" (UniqueName: \"kubernetes.io/projected/c66cf258-b98d-4b90-baa5-e557462560dd-kube-api-access-dftbk\") pod \"limitador-limitador-64c94fd64f-cdssm\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.498906 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.498875 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cdssm"] Jul 13 14:54:29.585710 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.585682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c66cf258-b98d-4b90-baa5-e557462560dd-config-file\") pod \"limitador-limitador-64c94fd64f-cdssm\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.585900 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.585745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftbk\" (UniqueName: \"kubernetes.io/projected/c66cf258-b98d-4b90-baa5-e557462560dd-kube-api-access-dftbk\") pod \"limitador-limitador-64c94fd64f-cdssm\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.586518 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.586487 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c66cf258-b98d-4b90-baa5-e557462560dd-config-file\") pod \"limitador-limitador-64c94fd64f-cdssm\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.597542 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.597512 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftbk\" (UniqueName: \"kubernetes.io/projected/c66cf258-b98d-4b90-baa5-e557462560dd-kube-api-access-dftbk\") pod \"limitador-limitador-64c94fd64f-cdssm\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.713653 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.713569 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:29.718878 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.718852 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:54:29.722727 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.722707 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:29.732325 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.732300 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:54:29.757955 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.757919 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:54:29.787240 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.787208 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6xw\" (UniqueName: \"kubernetes.io/projected/65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9-kube-api-access-cn6xw\") pod \"limitador-limitador-54d6d988d5-ndlnp\" (UID: \"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:29.787423 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.787312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9-config-file\") pod \"limitador-limitador-54d6d988d5-ndlnp\" (UID: \"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:29.864411 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.864292 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cdssm"] Jul 13 14:54:29.867285 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:54:29.867259 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66cf258_b98d_4b90_baa5_e557462560dd.slice/crio-a9c0225187c671ffe3e924f044156d908267276274f444df702b29227188a5ca WatchSource:0}: Error finding container a9c0225187c671ffe3e924f044156d908267276274f444df702b29227188a5ca: Status 404 returned error can't find the container with id a9c0225187c671ffe3e924f044156d908267276274f444df702b29227188a5ca Jul 13 14:54:29.888254 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.888230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6xw\" (UniqueName: \"kubernetes.io/projected/65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9-kube-api-access-cn6xw\") pod \"limitador-limitador-54d6d988d5-ndlnp\" (UID: \"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:29.888381 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.888289 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9-config-file\") pod \"limitador-limitador-54d6d988d5-ndlnp\" (UID: \"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:29.888903 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.888882 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9-config-file\") pod \"limitador-limitador-54d6d988d5-ndlnp\" (UID: \"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:29.897649 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:29.897624 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6xw\" (UniqueName: \"kubernetes.io/projected/65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9-kube-api-access-cn6xw\") pod \"limitador-limitador-54d6d988d5-ndlnp\" (UID: \"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:30.051722 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:30.051692 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:30.171731 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:30.171699 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" event={"ID":"c66cf258-b98d-4b90-baa5-e557462560dd","Type":"ContainerStarted","Data":"a9c0225187c671ffe3e924f044156d908267276274f444df702b29227188a5ca"} Jul 13 14:54:30.189097 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:30.189073 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:54:30.191574 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:54:30.191548 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d3e1e8_ce10_4f9e_9fe6_0d14d0909ef9.slice/crio-106415b0a63f826bfb2eb735e4a367ff90c272233c1068361be21be9c07adffc WatchSource:0}: Error finding container 106415b0a63f826bfb2eb735e4a367ff90c272233c1068361be21be9c07adffc: Status 404 returned error can't find the container with id 106415b0a63f826bfb2eb735e4a367ff90c272233c1068361be21be9c07adffc Jul 13 14:54:31.186595 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:31.186472 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" event={"ID":"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9","Type":"ContainerStarted","Data":"106415b0a63f826bfb2eb735e4a367ff90c272233c1068361be21be9c07adffc"} Jul 13 14:54:34.208787 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:34.208744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" event={"ID":"65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9","Type":"ContainerStarted","Data":"65d6c4ac7d4b33f8b1a89811614ee9bb6d64875fdcc1f50c20fde3f8a0c6a59d"} Jul 13 14:54:34.209250 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:34.208935 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:34.210193 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:34.210168 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" event={"ID":"c66cf258-b98d-4b90-baa5-e557462560dd","Type":"ContainerStarted","Data":"1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d"} Jul 13 14:54:34.210326 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:34.210307 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:34.227895 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:34.227852 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" podStartSLOduration=2.248708968 podStartE2EDuration="5.227838655s" totalImagesPullingTime="2.979129687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:34.2263497 +0000 UTC m=+511.880316156" watchObservedRunningTime="2026-07-13 14:54:34.227838655 +0000 UTC m=+511.881805175" Jul 13 14:54:45.216293 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.216262 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ndlnp" Jul 13 14:54:45.216698 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.216334 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:45.237322 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.237272 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" podStartSLOduration=12.946597054 podStartE2EDuration="16.237255907s" totalImagesPullingTime="3.290658853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:34.24705894 +0000 UTC m=+511.901025396" watchObservedRunningTime="2026-07-13 14:54:45.237255907 +0000 UTC m=+522.891222369" Jul 13 14:54:45.281412 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.281380 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cdssm"] Jul 13 14:54:45.281894 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.281850 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" podUID="c66cf258-b98d-4b90-baa5-e557462560dd" containerName="limitador" containerID="cri-o://1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d" gracePeriod=30 Jul 13 14:54:45.826834 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.826810 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:45.936019 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.935939 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dftbk\" (UniqueName: \"kubernetes.io/projected/c66cf258-b98d-4b90-baa5-e557462560dd-kube-api-access-dftbk\") pod \"c66cf258-b98d-4b90-baa5-e557462560dd\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " Jul 13 14:54:45.936212 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.936039 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c66cf258-b98d-4b90-baa5-e557462560dd-config-file\") pod \"c66cf258-b98d-4b90-baa5-e557462560dd\" (UID: \"c66cf258-b98d-4b90-baa5-e557462560dd\") " Jul 13 14:54:45.936552 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.936515 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66cf258-b98d-4b90-baa5-e557462560dd-config-file" (OuterVolumeSpecName: "config-file") pod "c66cf258-b98d-4b90-baa5-e557462560dd" (UID: "c66cf258-b98d-4b90-baa5-e557462560dd"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:45.938117 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:45.938096 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66cf258-b98d-4b90-baa5-e557462560dd-kube-api-access-dftbk" (OuterVolumeSpecName: "kube-api-access-dftbk") pod "c66cf258-b98d-4b90-baa5-e557462560dd" (UID: "c66cf258-b98d-4b90-baa5-e557462560dd"). InnerVolumeSpecName "kube-api-access-dftbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:46.037245 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.037202 2571 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c66cf258-b98d-4b90-baa5-e557462560dd-config-file\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:54:46.037245 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.037242 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dftbk\" (UniqueName: \"kubernetes.io/projected/c66cf258-b98d-4b90-baa5-e557462560dd-kube-api-access-dftbk\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:54:46.272515 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.272430 2571 generic.go:356] "Generic (PLEG): container finished" podID="c66cf258-b98d-4b90-baa5-e557462560dd" containerID="1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d" exitCode=0 Jul 13 14:54:46.272515 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.272477 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" Jul 13 14:54:46.272973 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.272515 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" event={"ID":"c66cf258-b98d-4b90-baa5-e557462560dd","Type":"ContainerDied","Data":"1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d"} Jul 13 14:54:46.272973 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.272567 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cdssm" event={"ID":"c66cf258-b98d-4b90-baa5-e557462560dd","Type":"ContainerDied","Data":"a9c0225187c671ffe3e924f044156d908267276274f444df702b29227188a5ca"} Jul 13 14:54:46.272973 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.272588 2571 scope.go:117] "RemoveContainer" containerID="1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d" Jul 13 14:54:46.281225 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.281207 2571 scope.go:117] "RemoveContainer" containerID="1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d" Jul 13 14:54:46.281470 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:54:46.281452 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d\": container with ID starting with 1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d not found: ID does not exist" containerID="1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d" Jul 13 14:54:46.281523 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.281477 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d"} err="failed to get container status \"1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d\": rpc error: code = NotFound desc = could not find container \"1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d\": container with ID starting with 1916bacc16cf15d2214c4ff56927f31954269e468b00f5b615713b96b202615d not found: ID does not exist" Jul 13 14:54:46.294059 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.294035 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cdssm"] Jul 13 14:54:46.296805 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.296785 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cdssm"] Jul 13 14:54:46.870194 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:54:46.870139 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66cf258-b98d-4b90-baa5-e557462560dd" path="/var/lib/kubelet/pods/c66cf258-b98d-4b90-baa5-e557462560dd/volumes" Jul 13 14:55:01.238619 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.238529 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tcgtv"] Jul 13 14:55:01.239146 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.239091 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66cf258-b98d-4b90-baa5-e557462560dd" containerName="limitador" Jul 13 14:55:01.239146 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.239111 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66cf258-b98d-4b90-baa5-e557462560dd" containerName="limitador" Jul 13 14:55:01.239273 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.239234 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66cf258-b98d-4b90-baa5-e557462560dd" containerName="limitador" Jul 13 14:55:01.242530 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.242511 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" Jul 13 14:55:01.245838 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.245773 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j2n7z\"" Jul 13 14:55:01.248194 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.248145 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tcgtv"] Jul 13 14:55:01.379125 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.379088 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9kr\" (UniqueName: \"kubernetes.io/projected/e16fbeec-606c-4bd3-a5c0-b67559c9c951-kube-api-access-4h9kr\") pod \"authorino-5d4b67c446-tcgtv\" (UID: \"e16fbeec-606c-4bd3-a5c0-b67559c9c951\") " pod="kuadrant-system/authorino-5d4b67c446-tcgtv" Jul 13 14:55:01.479947 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.479913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9kr\" (UniqueName: \"kubernetes.io/projected/e16fbeec-606c-4bd3-a5c0-b67559c9c951-kube-api-access-4h9kr\") pod \"authorino-5d4b67c446-tcgtv\" (UID: \"e16fbeec-606c-4bd3-a5c0-b67559c9c951\") " pod="kuadrant-system/authorino-5d4b67c446-tcgtv" Jul 13 14:55:01.493378 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.493311 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9kr\" (UniqueName: \"kubernetes.io/projected/e16fbeec-606c-4bd3-a5c0-b67559c9c951-kube-api-access-4h9kr\") pod \"authorino-5d4b67c446-tcgtv\" (UID: \"e16fbeec-606c-4bd3-a5c0-b67559c9c951\") " pod="kuadrant-system/authorino-5d4b67c446-tcgtv" Jul 13 14:55:01.530957 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.530925 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tcgtv"] Jul 13 14:55:01.531208 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.531196 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" Jul 13 14:55:01.560022 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.559984 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fd474bc48-klz66"] Jul 13 14:55:01.564470 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.564445 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:01.567296 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.567271 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 13 14:55:01.574694 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.574669 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fd474bc48-klz66"] Jul 13 14:55:01.665103 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.665081 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tcgtv"] Jul 13 14:55:01.667230 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:55:01.667197 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode16fbeec_606c_4bd3_a5c0_b67559c9c951.slice/crio-89754f728ceed5855d2b96812ed96e6608c7b154fba68e7735942580c6bd2c73 WatchSource:0}: Error finding container 89754f728ceed5855d2b96812ed96e6608c7b154fba68e7735942580c6bd2c73: Status 404 returned error can't find the container with id 89754f728ceed5855d2b96812ed96e6608c7b154fba68e7735942580c6bd2c73 Jul 13 14:55:01.681578 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.681551 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssp5\" (UniqueName: \"kubernetes.io/projected/944d7153-3c61-42fb-82eb-558c8fea4d44-kube-api-access-lssp5\") pod \"authorino-fd474bc48-klz66\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:01.681799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.681608 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/944d7153-3c61-42fb-82eb-558c8fea4d44-tls-cert\") pod \"authorino-fd474bc48-klz66\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:01.782991 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.782966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/944d7153-3c61-42fb-82eb-558c8fea4d44-tls-cert\") pod \"authorino-fd474bc48-klz66\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:01.783135 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.783040 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssp5\" (UniqueName: \"kubernetes.io/projected/944d7153-3c61-42fb-82eb-558c8fea4d44-kube-api-access-lssp5\") pod \"authorino-fd474bc48-klz66\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:01.785466 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.785444 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/944d7153-3c61-42fb-82eb-558c8fea4d44-tls-cert\") pod \"authorino-fd474bc48-klz66\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:01.792719 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.792695 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssp5\" (UniqueName: \"kubernetes.io/projected/944d7153-3c61-42fb-82eb-558c8fea4d44-kube-api-access-lssp5\") pod \"authorino-fd474bc48-klz66\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:01.875687 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:01.875657 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:55:02.013127 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:02.013101 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fd474bc48-klz66"] Jul 13 14:55:02.015044 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:55:02.015015 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod944d7153_3c61_42fb_82eb_558c8fea4d44.slice/crio-febfbba3b2e2dae6c72d4c39a66701cafd11da2f27026979ed8c0ec502a12316 WatchSource:0}: Error finding container febfbba3b2e2dae6c72d4c39a66701cafd11da2f27026979ed8c0ec502a12316: Status 404 returned error can't find the container with id febfbba3b2e2dae6c72d4c39a66701cafd11da2f27026979ed8c0ec502a12316 Jul 13 14:55:02.354458 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:02.354416 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd474bc48-klz66" event={"ID":"944d7153-3c61-42fb-82eb-558c8fea4d44","Type":"ContainerStarted","Data":"febfbba3b2e2dae6c72d4c39a66701cafd11da2f27026979ed8c0ec502a12316"} Jul 13 14:55:02.355632 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:02.355602 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" event={"ID":"e16fbeec-606c-4bd3-a5c0-b67559c9c951","Type":"ContainerStarted","Data":"89754f728ceed5855d2b96812ed96e6608c7b154fba68e7735942580c6bd2c73"} Jul 13 14:55:06.383780 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.383744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd474bc48-klz66" event={"ID":"944d7153-3c61-42fb-82eb-558c8fea4d44","Type":"ContainerStarted","Data":"9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338"} Jul 13 14:55:06.385318 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.385292 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" event={"ID":"e16fbeec-606c-4bd3-a5c0-b67559c9c951","Type":"ContainerStarted","Data":"560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94"} Jul 13 14:55:06.385420 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.385338 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" podUID="e16fbeec-606c-4bd3-a5c0-b67559c9c951" containerName="authorino" containerID="cri-o://560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94" gracePeriod=30 Jul 13 14:55:06.405244 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.405180 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fd474bc48-klz66" podStartSLOduration=1.768809902 podStartE2EDuration="5.405143132s" totalImagesPullingTime="3.63633323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:55:06.400932202 +0000 UTC m=+544.054898656" watchObservedRunningTime="2026-07-13 14:55:06.405143132 +0000 UTC m=+544.059109587" Jul 13 14:55:06.422431 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.422382 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" podStartSLOduration=1.451659945 podStartE2EDuration="5.422368045s" totalImagesPullingTime="3.9707081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:55:06.420922054 +0000 UTC m=+544.074888499" watchObservedRunningTime="2026-07-13 14:55:06.422368045 +0000 UTC m=+544.076334500" Jul 13 14:55:06.633738 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.633715 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" Jul 13 14:55:06.729788 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.729719 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h9kr\" (UniqueName: \"kubernetes.io/projected/e16fbeec-606c-4bd3-a5c0-b67559c9c951-kube-api-access-4h9kr\") pod \"e16fbeec-606c-4bd3-a5c0-b67559c9c951\" (UID: \"e16fbeec-606c-4bd3-a5c0-b67559c9c951\") " Jul 13 14:55:06.731715 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.731679 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16fbeec-606c-4bd3-a5c0-b67559c9c951-kube-api-access-4h9kr" (OuterVolumeSpecName: "kube-api-access-4h9kr") pod "e16fbeec-606c-4bd3-a5c0-b67559c9c951" (UID: "e16fbeec-606c-4bd3-a5c0-b67559c9c951"). InnerVolumeSpecName "kube-api-access-4h9kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:55:06.831296 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:06.831269 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9kr\" (UniqueName: \"kubernetes.io/projected/e16fbeec-606c-4bd3-a5c0-b67559c9c951-kube-api-access-4h9kr\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:55:07.392404 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.392367 2571 generic.go:356] "Generic (PLEG): container finished" podID="e16fbeec-606c-4bd3-a5c0-b67559c9c951" containerID="560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94" exitCode=0 Jul 13 14:55:07.392815 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.392409 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" Jul 13 14:55:07.392815 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.392453 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" event={"ID":"e16fbeec-606c-4bd3-a5c0-b67559c9c951","Type":"ContainerDied","Data":"560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94"} Jul 13 14:55:07.392815 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.392492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tcgtv" event={"ID":"e16fbeec-606c-4bd3-a5c0-b67559c9c951","Type":"ContainerDied","Data":"89754f728ceed5855d2b96812ed96e6608c7b154fba68e7735942580c6bd2c73"} Jul 13 14:55:07.392815 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.392508 2571 scope.go:117] "RemoveContainer" containerID="560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94" Jul 13 14:55:07.401037 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.401017 2571 scope.go:117] "RemoveContainer" containerID="560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94" Jul 13 14:55:07.401286 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:55:07.401267 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94\": container with ID starting with 560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94 not found: ID does not exist" containerID="560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94" Jul 13 14:55:07.401349 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.401294 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94"} err="failed to get container status \"560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94\": rpc error: code = NotFound desc = could not find container \"560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94\": container with ID starting with 560f631c166ec1c01ccec248434eab6cf58fe9fb82e279ee698dd9a89ea94a94 not found: ID does not exist" Jul 13 14:55:07.409672 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.409648 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tcgtv"] Jul 13 14:55:07.412028 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:07.412008 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tcgtv"] Jul 13 14:55:08.870546 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:08.870513 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16fbeec-606c-4bd3-a5c0-b67559c9c951" path="/var/lib/kubelet/pods/e16fbeec-606c-4bd3-a5c0-b67559c9c951/volumes" Jul 13 14:55:46.498927 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.498887 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:55:46.518206 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.518175 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-d46n7"] Jul 13 14:55:46.518565 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.518552 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16fbeec-606c-4bd3-a5c0-b67559c9c951" containerName="authorino" Jul 13 14:55:46.518608 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.518568 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16fbeec-606c-4bd3-a5c0-b67559c9c951" containerName="authorino" Jul 13 14:55:46.518642 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.518622 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16fbeec-606c-4bd3-a5c0-b67559c9c951" containerName="authorino" Jul 13 14:55:46.522863 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.522842 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.526399 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.526367 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 13 14:55:46.526571 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.526475 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-h4vz2\"" Jul 13 14:55:46.533735 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.533706 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-d46n7"] Jul 13 14:55:46.562278 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.562246 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-w56cq"] Jul 13 14:55:46.565056 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.565035 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.578217 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.578193 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-w56cq"] Jul 13 14:55:46.686058 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.686025 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92093db2-28be-4389-b673-32c7fe2f7b81-plugins-config-volume\") pod \"payload-processing-58f86c7798-d46n7\" (UID: \"92093db2-28be-4389-b673-32c7fe2f7b81\") " pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.686058 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.686057 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2z44\" (UniqueName: \"kubernetes.io/projected/92093db2-28be-4389-b673-32c7fe2f7b81-kube-api-access-l2z44\") pod \"payload-processing-58f86c7798-d46n7\" (UID: \"92093db2-28be-4389-b673-32c7fe2f7b81\") " pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.686323 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.686180 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e715bff4-309b-42fb-87af-0c7d81a0a5e6-plugins-config-volume\") pod \"payload-pre-processing-7964457696-w56cq\" (UID: \"e715bff4-309b-42fb-87af-0c7d81a0a5e6\") " pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.686323 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.686239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrmh\" (UniqueName: \"kubernetes.io/projected/e715bff4-309b-42fb-87af-0c7d81a0a5e6-kube-api-access-pkrmh\") pod \"payload-pre-processing-7964457696-w56cq\" (UID: \"e715bff4-309b-42fb-87af-0c7d81a0a5e6\") " pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.787236 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.787201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92093db2-28be-4389-b673-32c7fe2f7b81-plugins-config-volume\") pod \"payload-processing-58f86c7798-d46n7\" (UID: \"92093db2-28be-4389-b673-32c7fe2f7b81\") " pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.787397 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.787243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2z44\" (UniqueName: \"kubernetes.io/projected/92093db2-28be-4389-b673-32c7fe2f7b81-kube-api-access-l2z44\") pod \"payload-processing-58f86c7798-d46n7\" (UID: \"92093db2-28be-4389-b673-32c7fe2f7b81\") " pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.787397 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.787295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e715bff4-309b-42fb-87af-0c7d81a0a5e6-plugins-config-volume\") pod \"payload-pre-processing-7964457696-w56cq\" (UID: \"e715bff4-309b-42fb-87af-0c7d81a0a5e6\") " pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.787397 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.787334 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrmh\" (UniqueName: \"kubernetes.io/projected/e715bff4-309b-42fb-87af-0c7d81a0a5e6-kube-api-access-pkrmh\") pod \"payload-pre-processing-7964457696-w56cq\" (UID: \"e715bff4-309b-42fb-87af-0c7d81a0a5e6\") " pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.787956 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.787936 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e715bff4-309b-42fb-87af-0c7d81a0a5e6-plugins-config-volume\") pod \"payload-pre-processing-7964457696-w56cq\" (UID: \"e715bff4-309b-42fb-87af-0c7d81a0a5e6\") " pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.788029 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.788009 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92093db2-28be-4389-b673-32c7fe2f7b81-plugins-config-volume\") pod \"payload-processing-58f86c7798-d46n7\" (UID: \"92093db2-28be-4389-b673-32c7fe2f7b81\") " pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.801903 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.801875 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrmh\" (UniqueName: \"kubernetes.io/projected/e715bff4-309b-42fb-87af-0c7d81a0a5e6-kube-api-access-pkrmh\") pod \"payload-pre-processing-7964457696-w56cq\" (UID: \"e715bff4-309b-42fb-87af-0c7d81a0a5e6\") " pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.802879 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.802853 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2z44\" (UniqueName: \"kubernetes.io/projected/92093db2-28be-4389-b673-32c7fe2f7b81-kube-api-access-l2z44\") pod \"payload-processing-58f86c7798-d46n7\" (UID: \"92093db2-28be-4389-b673-32c7fe2f7b81\") " pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.834657 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.834635 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:46.876773 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.876737 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:46.988545 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:46.988517 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-d46n7"] Jul 13 14:55:46.994049 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:55:46.994021 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92093db2_28be_4389_b673_32c7fe2f7b81.slice/crio-1d83b7356cd89dd18b979c7622c17377f266ebc0453d52fe6ba202c49e544bac WatchSource:0}: Error finding container 1d83b7356cd89dd18b979c7622c17377f266ebc0453d52fe6ba202c49e544bac: Status 404 returned error can't find the container with id 1d83b7356cd89dd18b979c7622c17377f266ebc0453d52fe6ba202c49e544bac Jul 13 14:55:47.050798 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:47.050639 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-w56cq"] Jul 13 14:55:47.063176 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:55:47.062454 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode715bff4_309b_42fb_87af_0c7d81a0a5e6.slice/crio-9bb3fcf2161abfc6a7b13a544a9aedb358938aeb46f9e64088c5141427a84269 WatchSource:0}: Error finding container 9bb3fcf2161abfc6a7b13a544a9aedb358938aeb46f9e64088c5141427a84269: Status 404 returned error can't find the container with id 9bb3fcf2161abfc6a7b13a544a9aedb358938aeb46f9e64088c5141427a84269 Jul 13 14:55:47.582819 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:47.582782 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" event={"ID":"e715bff4-309b-42fb-87af-0c7d81a0a5e6","Type":"ContainerStarted","Data":"9bb3fcf2161abfc6a7b13a544a9aedb358938aeb46f9e64088c5141427a84269"} Jul 13 14:55:47.583975 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:47.583941 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-d46n7" event={"ID":"92093db2-28be-4389-b673-32c7fe2f7b81","Type":"ContainerStarted","Data":"1d83b7356cd89dd18b979c7622c17377f266ebc0453d52fe6ba202c49e544bac"} Jul 13 14:55:49.598333 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:49.598285 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-d46n7" event={"ID":"92093db2-28be-4389-b673-32c7fe2f7b81","Type":"ContainerStarted","Data":"85f54148dd2dd84b4ea816a9b46338d55086ac101f7922d33cd8f5097608de46"} Jul 13 14:55:49.598799 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:49.598367 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:55:49.599725 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:49.599702 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" event={"ID":"e715bff4-309b-42fb-87af-0c7d81a0a5e6","Type":"ContainerStarted","Data":"feb126565828a292e429cb815e9d5c3029c58683e1dab59bf0b7d6672fd42dde"} Jul 13 14:55:49.599852 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:49.599751 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:55:49.616282 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:49.616201 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-d46n7" podStartSLOduration=1.73053111 podStartE2EDuration="3.616183229s" totalImagesPullingTime="1.885652119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:55:49.615276502 +0000 UTC m=+587.269242956" watchObservedRunningTime="2026-07-13 14:55:49.616183229 +0000 UTC m=+587.270149682" Jul 13 14:55:49.631110 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:49.631056 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" podStartSLOduration=1.811909029 podStartE2EDuration="3.631037991s" totalImagesPullingTime="1.819128962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:55:49.630264805 +0000 UTC m=+587.284231271" watchObservedRunningTime="2026-07-13 14:55:49.631037991 +0000 UTC m=+587.285004450" Jul 13 14:55:53.173309 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:53.173272 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sjhsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 14:55:53.173679 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:53.173332 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" podUID="9280c649-ece0-4e47-be43-cca2b81c54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 14:55:54.629667 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:54.629631 2571 generic.go:356] "Generic (PLEG): container finished" podID="9280c649-ece0-4e47-be43-cca2b81c54b1" containerID="26d1bd67d9a1a2ef867939e12568ce3ec82ca30c5e555fa0b493aedddccb8332" exitCode=0 Jul 13 14:55:54.630211 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:54.629708 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" event={"ID":"9280c649-ece0-4e47-be43-cca2b81c54b1","Type":"ContainerDied","Data":"26d1bd67d9a1a2ef867939e12568ce3ec82ca30c5e555fa0b493aedddccb8332"} Jul 13 14:55:54.630294 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:54.630225 2571 scope.go:117] "RemoveContainer" containerID="26d1bd67d9a1a2ef867939e12568ce3ec82ca30c5e555fa0b493aedddccb8332" Jul 13 14:55:54.631263 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:54.631233 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:55:54.631346 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:54.631302 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:55:54.631346 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:54.631334 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 14:55:55.636851 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:55.636813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" event={"ID":"9280c649-ece0-4e47-be43-cca2b81c54b1","Type":"ContainerStarted","Data":"9dab0bad705613e57ee7de1a6dfcf2519d5e8d6397e9c0140a296b4930d74b2c"} Jul 13 14:55:56.084612 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:56.084570 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:55:56.084932 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:56.084909 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sjhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 13 14:55:56.085033 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:56.084964 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" podUID="9280c649-ece0-4e47-be43-cca2b81c54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 13 14:55:57.084665 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:57.084629 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sjhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 13 14:55:57.085064 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:57.084692 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" podUID="9280c649-ece0-4e47-be43-cca2b81c54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 13 14:55:58.085495 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:58.085460 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sjhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 13 14:55:58.085871 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:58.085529 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" podUID="9280c649-ece0-4e47-be43-cca2b81c54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 13 14:55:59.085143 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:59.085100 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sjhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 13 14:55:59.085374 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:55:59.085194 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" podUID="9280c649-ece0-4e47-be43-cca2b81c54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 13 14:56:00.085664 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:00.085620 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sjhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 13 14:56:00.086065 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:00.085695 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" podUID="9280c649-ece0-4e47-be43-cca2b81c54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 13 14:56:00.606469 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:00.606437 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-d46n7" Jul 13 14:56:00.606637 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:00.606508 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-w56cq" Jul 13 14:56:01.088759 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:01.088727 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:56:01.089189 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:01.088991 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:56:01.089686 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:01.089667 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sjhsx" Jul 13 14:56:02.802698 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:02.802669 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 14:56:02.803109 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:02.802902 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 14:56:04.623860 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:04.623822 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:56:08.991839 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:08.991809 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h"] Jul 13 14:56:08.996792 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:08.996772 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:08.999357 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:08.999331 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 14:56:09.000240 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.000221 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8467h\"" Jul 13 14:56:09.000348 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.000241 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 14:56:09.000348 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.000261 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 13 14:56:09.006023 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.006001 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h"] Jul 13 14:56:09.090637 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.090600 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.090814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.090646 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hkzz\" (UniqueName: \"kubernetes.io/projected/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-kube-api-access-5hkzz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.090814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.090706 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.090814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.090722 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.090814 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.090739 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.090986 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.090810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.191776 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.191735 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.191965 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.191790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hkzz\" (UniqueName: \"kubernetes.io/projected/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-kube-api-access-5hkzz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.191965 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.191837 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.191965 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.191859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.191965 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.191883 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.191965 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.191919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.192247 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.192224 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.192247 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.192241 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.192390 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.192370 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.194115 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.194099 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.194417 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.194401 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.201098 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.201069 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hkzz\" (UniqueName: \"kubernetes.io/projected/775580ba-d3ce-4fa8-864e-aaaf3ffd72ec-kube-api-access-5hkzz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-85m8h\" (UID: \"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.307964 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.307922 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:09.450188 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.450142 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h"] Jul 13 14:56:09.452744 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:56:09.452713 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775580ba_d3ce_4fa8_864e_aaaf3ffd72ec.slice/crio-6a273250a39e1a459257ae472f687fdeb2e993e2b9190964a6fcbed7ea8614f9 WatchSource:0}: Error finding container 6a273250a39e1a459257ae472f687fdeb2e993e2b9190964a6fcbed7ea8614f9: Status 404 returned error can't find the container with id 6a273250a39e1a459257ae472f687fdeb2e993e2b9190964a6fcbed7ea8614f9 Jul 13 14:56:09.713903 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.713821 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" event={"ID":"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec","Type":"ContainerStarted","Data":"6a273250a39e1a459257ae472f687fdeb2e993e2b9190964a6fcbed7ea8614f9"} Jul 13 14:56:09.906473 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:09.902811 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:56:11.725907 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:11.725850 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" event={"ID":"775580ba-d3ce-4fa8-864e-aaaf3ffd72ec","Type":"ContainerStarted","Data":"b61d29fca32fc281b14859aab733153e56a5fb77ea46a63a551f0358a41bda5d"} Jul 13 14:56:11.744283 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:11.744226 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" podStartSLOduration=2.517446465 podStartE2EDuration="3.744207247s" totalImagesPullingTime="1.226760782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:56:11.743980626 +0000 UTC m=+609.397947079" watchObservedRunningTime="2026-07-13 14:56:11.744207247 +0000 UTC m=+609.398173700" Jul 13 14:56:17.925338 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:17.925305 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6"] Jul 13 14:56:17.929021 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:17.929004 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:17.936684 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:17.936661 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 13 14:56:17.944296 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:17.944270 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6"] Jul 13 14:56:18.078667 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.078631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzbc\" (UniqueName: \"kubernetes.io/projected/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-kube-api-access-zvzbc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.078844 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.078698 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.078844 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.078775 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.078844 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.078826 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.079024 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.078856 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.079024 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.078900 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.179808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.179705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzbc\" (UniqueName: \"kubernetes.io/projected/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-kube-api-access-zvzbc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.179808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.179774 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.179808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.179793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.179808 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.179809 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.180187 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.179829 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.180187 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.179859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.180311 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.180270 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.180311 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.180281 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.180395 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.180317 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.182164 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.182130 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.182503 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.182485 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.188533 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.188512 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzbc\" (UniqueName: \"kubernetes.io/projected/91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e-kube-api-access-zvzbc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sl2j6\" (UID: \"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.239826 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.239791 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:18.382437 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.382414 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6"] Jul 13 14:56:18.385018 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:56:18.384989 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f0d4d0_e7ef_4078_a208_a0d8f24f8a2e.slice/crio-5c069822c58e74f0ded790ca12992c96a4656c7eca2ec1fc18ed0bb644053a39 WatchSource:0}: Error finding container 5c069822c58e74f0ded790ca12992c96a4656c7eca2ec1fc18ed0bb644053a39: Status 404 returned error can't find the container with id 5c069822c58e74f0ded790ca12992c96a4656c7eca2ec1fc18ed0bb644053a39 Jul 13 14:56:18.386487 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.386463 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:56:18.761595 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.761559 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" event={"ID":"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e","Type":"ContainerStarted","Data":"11e57365fb1a25880c3bbe1f837172889633b404a5349c51f4f8892ceb90a70f"} Jul 13 14:56:18.761765 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:18.761607 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" event={"ID":"91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e","Type":"ContainerStarted","Data":"5c069822c58e74f0ded790ca12992c96a4656c7eca2ec1fc18ed0bb644053a39"} Jul 13 14:56:19.096876 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:19.096823 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" podStartSLOduration=1.911619988 podStartE2EDuration="2.096798386s" totalImagesPullingTime="185.178398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:56:18.780778641 +0000 UTC m=+616.434745096" watchObservedRunningTime="2026-07-13 14:56:19.096798386 +0000 UTC m=+616.750764843" Jul 13 14:56:19.098467 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:19.098445 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:56:19.308419 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:19.308382 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:19.308579 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:19.308459 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:19.321747 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:19.321719 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:19.785598 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:19.785568 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-85m8h" Jul 13 14:56:26.000355 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:26.000267 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:56:28.240796 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:28.240758 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:28.241211 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:28.240834 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:28.253803 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:28.253775 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:28.828336 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:28.828306 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sl2j6" Jul 13 14:56:34.606922 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:34.606883 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:56:55.195073 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:56:55.195036 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:57:48.376738 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.376695 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fdd6798bf-g7lst"] Jul 13 14:57:48.380621 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.380595 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.391944 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.391917 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fdd6798bf-g7lst"] Jul 13 14:57:48.498983 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.498944 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4cc28573-d530-4efc-8c71-d935fe5a937e-tls-cert\") pod \"authorino-fdd6798bf-g7lst\" (UID: \"4cc28573-d530-4efc-8c71-d935fe5a937e\") " pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.498983 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.498987 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldj4x\" (UniqueName: \"kubernetes.io/projected/4cc28573-d530-4efc-8c71-d935fe5a937e-kube-api-access-ldj4x\") pod \"authorino-fdd6798bf-g7lst\" (UID: \"4cc28573-d530-4efc-8c71-d935fe5a937e\") " pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.600006 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.599965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4cc28573-d530-4efc-8c71-d935fe5a937e-tls-cert\") pod \"authorino-fdd6798bf-g7lst\" (UID: \"4cc28573-d530-4efc-8c71-d935fe5a937e\") " pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.600006 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.600011 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldj4x\" (UniqueName: \"kubernetes.io/projected/4cc28573-d530-4efc-8c71-d935fe5a937e-kube-api-access-ldj4x\") pod \"authorino-fdd6798bf-g7lst\" (UID: \"4cc28573-d530-4efc-8c71-d935fe5a937e\") " pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.602545 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.602522 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4cc28573-d530-4efc-8c71-d935fe5a937e-tls-cert\") pod \"authorino-fdd6798bf-g7lst\" (UID: \"4cc28573-d530-4efc-8c71-d935fe5a937e\") " pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.609220 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.609194 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldj4x\" (UniqueName: \"kubernetes.io/projected/4cc28573-d530-4efc-8c71-d935fe5a937e-kube-api-access-ldj4x\") pod \"authorino-fdd6798bf-g7lst\" (UID: \"4cc28573-d530-4efc-8c71-d935fe5a937e\") " pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.690105 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.690012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fdd6798bf-g7lst" Jul 13 14:57:48.834714 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:48.834682 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fdd6798bf-g7lst"] Jul 13 14:57:48.836439 ip-10-0-133-41 kubenswrapper[2571]: W0713 14:57:48.836408 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cc28573_d530_4efc_8c71_d935fe5a937e.slice/crio-bfa0cc9febfe466e0c12f726653865cb003a53b206cadf71e2bfe9626d12effa WatchSource:0}: Error finding container bfa0cc9febfe466e0c12f726653865cb003a53b206cadf71e2bfe9626d12effa: Status 404 returned error can't find the container with id bfa0cc9febfe466e0c12f726653865cb003a53b206cadf71e2bfe9626d12effa Jul 13 14:57:49.222766 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:49.222724 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fdd6798bf-g7lst" event={"ID":"4cc28573-d530-4efc-8c71-d935fe5a937e","Type":"ContainerStarted","Data":"bfa0cc9febfe466e0c12f726653865cb003a53b206cadf71e2bfe9626d12effa"} Jul 13 14:57:50.230050 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.230005 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fdd6798bf-g7lst" event={"ID":"4cc28573-d530-4efc-8c71-d935fe5a937e","Type":"ContainerStarted","Data":"c1ff400e25c9dc0b135661e791104be75fc97ea6d9d1bdd5e9734d50f1d0058d"} Jul 13 14:57:50.247476 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.247420 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fdd6798bf-g7lst" podStartSLOduration=1.814290446 podStartE2EDuration="2.247402138s" totalImagesPullingTime="433.111692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:57:50.245509106 +0000 UTC m=+707.899475562" watchObservedRunningTime="2026-07-13 14:57:50.247402138 +0000 UTC m=+707.901368759" Jul 13 14:57:50.280547 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.280508 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fd474bc48-klz66"] Jul 13 14:57:50.280760 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.280729 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-fd474bc48-klz66" podUID="944d7153-3c61-42fb-82eb-558c8fea4d44" containerName="authorino" containerID="cri-o://9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338" gracePeriod=30 Jul 13 14:57:50.552894 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.552867 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:57:50.622954 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.622917 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/944d7153-3c61-42fb-82eb-558c8fea4d44-tls-cert\") pod \"944d7153-3c61-42fb-82eb-558c8fea4d44\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " Jul 13 14:57:50.623142 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.623041 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lssp5\" (UniqueName: \"kubernetes.io/projected/944d7153-3c61-42fb-82eb-558c8fea4d44-kube-api-access-lssp5\") pod \"944d7153-3c61-42fb-82eb-558c8fea4d44\" (UID: \"944d7153-3c61-42fb-82eb-558c8fea4d44\") " Jul 13 14:57:50.625342 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.625306 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944d7153-3c61-42fb-82eb-558c8fea4d44-kube-api-access-lssp5" (OuterVolumeSpecName: "kube-api-access-lssp5") pod "944d7153-3c61-42fb-82eb-558c8fea4d44" (UID: "944d7153-3c61-42fb-82eb-558c8fea4d44"). InnerVolumeSpecName "kube-api-access-lssp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:57:50.633851 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.633824 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944d7153-3c61-42fb-82eb-558c8fea4d44-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "944d7153-3c61-42fb-82eb-558c8fea4d44" (UID: "944d7153-3c61-42fb-82eb-558c8fea4d44"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:57:50.724631 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.724597 2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/944d7153-3c61-42fb-82eb-558c8fea4d44-tls-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.724631 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:50.724629 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lssp5\" (UniqueName: \"kubernetes.io/projected/944d7153-3c61-42fb-82eb-558c8fea4d44-kube-api-access-lssp5\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 14:57:51.237811 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.237726 2571 generic.go:356] "Generic (PLEG): container finished" podID="944d7153-3c61-42fb-82eb-558c8fea4d44" containerID="9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338" exitCode=0 Jul 13 14:57:51.237811 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.237772 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd474bc48-klz66" Jul 13 14:57:51.238367 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.237806 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd474bc48-klz66" event={"ID":"944d7153-3c61-42fb-82eb-558c8fea4d44","Type":"ContainerDied","Data":"9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338"} Jul 13 14:57:51.238367 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.237851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd474bc48-klz66" event={"ID":"944d7153-3c61-42fb-82eb-558c8fea4d44","Type":"ContainerDied","Data":"febfbba3b2e2dae6c72d4c39a66701cafd11da2f27026979ed8c0ec502a12316"} Jul 13 14:57:51.238367 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.237876 2571 scope.go:117] "RemoveContainer" containerID="9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338" Jul 13 14:57:51.247261 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.247242 2571 scope.go:117] "RemoveContainer" containerID="9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338" Jul 13 14:57:51.247511 ip-10-0-133-41 kubenswrapper[2571]: E0713 14:57:51.247490 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338\": container with ID starting with 9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338 not found: ID does not exist" containerID="9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338" Jul 13 14:57:51.247621 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.247518 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338"} err="failed to get container status \"9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338\": rpc error: code = NotFound desc = could not find container \"9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338\": container with ID starting with 9ab93fb3d386f7e0c34f87711ae3f41fe6646dbe2c34a2333e16995b95fb7338 not found: ID does not exist" Jul 13 14:57:51.256529 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.256502 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fd474bc48-klz66"] Jul 13 14:57:51.258864 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:51.258839 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-fd474bc48-klz66"] Jul 13 14:57:52.871162 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:57:52.871129 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="944d7153-3c61-42fb-82eb-558c8fea4d44" path="/var/lib/kubelet/pods/944d7153-3c61-42fb-82eb-558c8fea4d44/volumes" Jul 13 14:58:18.697292 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:58:18.697258 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:58:28.203864 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:58:28.203820 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:58:40.198096 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:58:40.198059 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:58:48.601786 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:58:48.601749 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:58:59.498703 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:58:59.498657 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:59:26.207043 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:59:26.206936 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 14:59:38.294871 ip-10-0-133-41 kubenswrapper[2571]: I0713 14:59:38.294833 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:00:00.189391 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.189354 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29732580-pzndm"] Jul 13 15:00:00.189840 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.189758 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944d7153-3c61-42fb-82eb-558c8fea4d44" containerName="authorino" Jul 13 15:00:00.189840 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.189769 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="944d7153-3c61-42fb-82eb-558c8fea4d44" containerName="authorino" Jul 13 15:00:00.189913 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.189848 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="944d7153-3c61-42fb-82eb-558c8fea4d44" containerName="authorino" Jul 13 15:00:00.192791 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.192769 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" Jul 13 15:00:00.195172 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.195135 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29732580-pzndm"] Jul 13 15:00:00.196438 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.196420 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9vf7z\"" Jul 13 15:00:00.229236 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.229196 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwj9\" (UniqueName: \"kubernetes.io/projected/c3c1834c-3e8d-4f59-9d3c-59a2daff9a47-kube-api-access-nlwj9\") pod \"maas-api-key-cleanup-29732580-pzndm\" (UID: \"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47\") " pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" Jul 13 15:00:00.330193 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.330136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwj9\" (UniqueName: \"kubernetes.io/projected/c3c1834c-3e8d-4f59-9d3c-59a2daff9a47-kube-api-access-nlwj9\") pod \"maas-api-key-cleanup-29732580-pzndm\" (UID: \"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47\") " pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" Jul 13 15:00:00.339723 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.339701 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwj9\" (UniqueName: \"kubernetes.io/projected/c3c1834c-3e8d-4f59-9d3c-59a2daff9a47-kube-api-access-nlwj9\") pod \"maas-api-key-cleanup-29732580-pzndm\" (UID: \"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47\") " pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" Jul 13 15:00:00.504266 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.504148 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" Jul 13 15:00:00.855528 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.855499 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29732580-pzndm"] Jul 13 15:00:00.858318 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:00:00.858254 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c1834c_3e8d_4f59_9d3c_59a2daff9a47.slice/crio-cf40b73ca411d6a7ffadeb1720009116b8769489cc958dd39b678f99d614795e WatchSource:0}: Error finding container cf40b73ca411d6a7ffadeb1720009116b8769489cc958dd39b678f99d614795e: Status 404 returned error can't find the container with id cf40b73ca411d6a7ffadeb1720009116b8769489cc958dd39b678f99d614795e Jul 13 15:00:00.899279 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:00.899243 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" event={"ID":"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47","Type":"ContainerStarted","Data":"cf40b73ca411d6a7ffadeb1720009116b8769489cc958dd39b678f99d614795e"} Jul 13 15:00:16.808256 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:16.808218 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:00:25.038809 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:25.038775 2571 generic.go:356] "Generic (PLEG): container finished" podID="c3c1834c-3e8d-4f59-9d3c-59a2daff9a47" containerID="76322e31864c822262193337b0cf69df9ef343f7e0f9092defdb4db85e7c3f89" exitCode=0 Jul 13 15:00:25.039254 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:25.038863 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" event={"ID":"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47","Type":"ContainerDied","Data":"76322e31864c822262193337b0cf69df9ef343f7e0f9092defdb4db85e7c3f89"} Jul 13 15:00:26.182844 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:26.182815 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" Jul 13 15:00:26.278941 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:26.278905 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlwj9\" (UniqueName: \"kubernetes.io/projected/c3c1834c-3e8d-4f59-9d3c-59a2daff9a47-kube-api-access-nlwj9\") pod \"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47\" (UID: \"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47\") " Jul 13 15:00:26.281134 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:26.281106 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c1834c-3e8d-4f59-9d3c-59a2daff9a47-kube-api-access-nlwj9" (OuterVolumeSpecName: "kube-api-access-nlwj9") pod "c3c1834c-3e8d-4f59-9d3c-59a2daff9a47" (UID: "c3c1834c-3e8d-4f59-9d3c-59a2daff9a47"). InnerVolumeSpecName "kube-api-access-nlwj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:26.379647 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:26.379563 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlwj9\" (UniqueName: \"kubernetes.io/projected/c3c1834c-3e8d-4f59-9d3c-59a2daff9a47-kube-api-access-nlwj9\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:00:26.599144 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:26.599110 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:00:27.051841 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:27.051807 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" event={"ID":"c3c1834c-3e8d-4f59-9d3c-59a2daff9a47","Type":"ContainerDied","Data":"cf40b73ca411d6a7ffadeb1720009116b8769489cc958dd39b678f99d614795e"} Jul 13 15:00:27.052037 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:27.051859 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf40b73ca411d6a7ffadeb1720009116b8769489cc958dd39b678f99d614795e" Jul 13 15:00:27.052037 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:27.051911 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732580-pzndm" Jul 13 15:00:35.696809 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:35.696777 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:00:46.908452 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:46.908416 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:00:57.693071 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:00:57.692973 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:01:02.834593 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:02.834559 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:01:02.835983 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:02.835948 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:01:07.221765 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:07.221725 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:01:19.417853 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:19.417814 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:01:26.800164 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:26.800113 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:01:34.204420 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:34.204376 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:01:41.919964 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:41.919928 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:01:54.305915 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:01:54.305880 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:02:05.707810 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:02:05.707773 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:02:30.311419 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:02:30.311323 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:02:40.795031 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:02:40.794997 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:02:56.598928 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:02:56.598888 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:03:07.215386 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:03:07.215348 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:03:16.605322 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:03:16.605285 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:03:26.507167 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:03:26.507129 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:03:35.597414 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:03:35.597381 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:04:02.437520 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:02.437430 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l"] Jul 13 15:04:02.438105 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:02.437666 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" podUID="85d4459c-e569-4ce4-aa24-2335440533da" containerName="manager" containerID="cri-o://346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd" gracePeriod=10 Jul 13 15:04:05.798194 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:05.798116 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 15:04:05.895125 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:05.895087 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d4459c-e569-4ce4-aa24-2335440533da-extensions-socket-volume\") pod \"85d4459c-e569-4ce4-aa24-2335440533da\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " Jul 13 15:04:05.895344 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:05.895141 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rnqc\" (UniqueName: \"kubernetes.io/projected/85d4459c-e569-4ce4-aa24-2335440533da-kube-api-access-9rnqc\") pod \"85d4459c-e569-4ce4-aa24-2335440533da\" (UID: \"85d4459c-e569-4ce4-aa24-2335440533da\") " Jul 13 15:04:05.895504 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:05.895478 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d4459c-e569-4ce4-aa24-2335440533da-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "85d4459c-e569-4ce4-aa24-2335440533da" (UID: "85d4459c-e569-4ce4-aa24-2335440533da"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:04:05.897162 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:05.897126 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d4459c-e569-4ce4-aa24-2335440533da-kube-api-access-9rnqc" (OuterVolumeSpecName: "kube-api-access-9rnqc") pod "85d4459c-e569-4ce4-aa24-2335440533da" (UID: "85d4459c-e569-4ce4-aa24-2335440533da"). InnerVolumeSpecName "kube-api-access-9rnqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:04:05.996366 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:05.996332 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rnqc\" (UniqueName: \"kubernetes.io/projected/85d4459c-e569-4ce4-aa24-2335440533da-kube-api-access-9rnqc\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:04:05.996366 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:05.996358 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d4459c-e569-4ce4-aa24-2335440533da-extensions-socket-volume\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:04:06.215045 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.214957 2571 generic.go:356] "Generic (PLEG): container finished" podID="85d4459c-e569-4ce4-aa24-2335440533da" containerID="346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd" exitCode=0 Jul 13 15:04:06.215045 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.214997 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" Jul 13 15:04:06.215045 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.215031 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" event={"ID":"85d4459c-e569-4ce4-aa24-2335440533da","Type":"ContainerDied","Data":"346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd"} Jul 13 15:04:06.215299 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.215081 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l" event={"ID":"85d4459c-e569-4ce4-aa24-2335440533da","Type":"ContainerDied","Data":"c4f4a04baa37a6e7d530e62442e31e8747896334b0f94e322c4a2914b590f857"} Jul 13 15:04:06.215299 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.215105 2571 scope.go:117] "RemoveContainer" containerID="346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd" Jul 13 15:04:06.225742 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.225723 2571 scope.go:117] "RemoveContainer" containerID="346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd" Jul 13 15:04:06.225988 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:04:06.225970 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd\": container with ID starting with 346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd not found: ID does not exist" containerID="346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd" Jul 13 15:04:06.226040 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.225998 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd"} err="failed to get container status \"346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd\": rpc error: code = NotFound desc = could not find container \"346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd\": container with ID starting with 346eb0dfbd1681ddd74172a66a8fb591d2ffef3953b7b0cd5d06e407ec7436bd not found: ID does not exist" Jul 13 15:04:06.241460 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.241438 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l"] Jul 13 15:04:06.248339 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.248316 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tnv9l"] Jul 13 15:04:06.870662 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:04:06.870626 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d4459c-e569-4ce4-aa24-2335440533da" path="/var/lib/kubelet/pods/85d4459c-e569-4ce4-aa24-2335440533da/volumes" Jul 13 15:05:03.701633 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.701591 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s"] Jul 13 15:05:03.702060 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.702029 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c1834c-3e8d-4f59-9d3c-59a2daff9a47" containerName="cleanup" Jul 13 15:05:03.702060 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.702042 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c1834c-3e8d-4f59-9d3c-59a2daff9a47" containerName="cleanup" Jul 13 15:05:03.702060 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.702059 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d4459c-e569-4ce4-aa24-2335440533da" containerName="manager" Jul 13 15:05:03.702186 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.702066 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d4459c-e569-4ce4-aa24-2335440533da" containerName="manager" Jul 13 15:05:03.702186 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.702127 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d4459c-e569-4ce4-aa24-2335440533da" containerName="manager" Jul 13 15:05:03.702186 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.702141 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c1834c-3e8d-4f59-9d3c-59a2daff9a47" containerName="cleanup" Jul 13 15:05:03.706503 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.706480 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:03.709396 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.709375 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qvsps\"" Jul 13 15:05:03.716465 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.716437 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s"] Jul 13 15:05:03.823886 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.823846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/973ed3e2-0703-47d1-96c8-3c2ae7541433-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qhs4s\" (UID: \"973ed3e2-0703-47d1-96c8-3c2ae7541433\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:03.824110 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.823963 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762kh\" (UniqueName: \"kubernetes.io/projected/973ed3e2-0703-47d1-96c8-3c2ae7541433-kube-api-access-762kh\") pod \"kuadrant-operator-controller-manager-86878c769f-qhs4s\" (UID: \"973ed3e2-0703-47d1-96c8-3c2ae7541433\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:03.924598 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.924553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/973ed3e2-0703-47d1-96c8-3c2ae7541433-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qhs4s\" (UID: \"973ed3e2-0703-47d1-96c8-3c2ae7541433\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:03.924775 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.924630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762kh\" (UniqueName: \"kubernetes.io/projected/973ed3e2-0703-47d1-96c8-3c2ae7541433-kube-api-access-762kh\") pod \"kuadrant-operator-controller-manager-86878c769f-qhs4s\" (UID: \"973ed3e2-0703-47d1-96c8-3c2ae7541433\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:03.924944 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.924922 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/973ed3e2-0703-47d1-96c8-3c2ae7541433-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qhs4s\" (UID: \"973ed3e2-0703-47d1-96c8-3c2ae7541433\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:03.935612 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:03.935577 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-762kh\" (UniqueName: \"kubernetes.io/projected/973ed3e2-0703-47d1-96c8-3c2ae7541433-kube-api-access-762kh\") pod \"kuadrant-operator-controller-manager-86878c769f-qhs4s\" (UID: \"973ed3e2-0703-47d1-96c8-3c2ae7541433\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:04.018873 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:04.018783 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:04.167927 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:04.167879 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s"] Jul 13 15:05:04.171463 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:05:04.171433 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973ed3e2_0703_47d1_96c8_3c2ae7541433.slice/crio-eedd81fcd388a4e3c95e8bca3f732a49600a350217b8988afa8ce147cf61ea5e WatchSource:0}: Error finding container eedd81fcd388a4e3c95e8bca3f732a49600a350217b8988afa8ce147cf61ea5e: Status 404 returned error can't find the container with id eedd81fcd388a4e3c95e8bca3f732a49600a350217b8988afa8ce147cf61ea5e Jul 13 15:05:04.503508 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:04.503476 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" event={"ID":"973ed3e2-0703-47d1-96c8-3c2ae7541433","Type":"ContainerStarted","Data":"76a25e6e415c261c0a616b557a298c0d680059144756f41848b4e840750f5feb"} Jul 13 15:05:04.503707 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:04.503529 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" event={"ID":"973ed3e2-0703-47d1-96c8-3c2ae7541433","Type":"ContainerStarted","Data":"eedd81fcd388a4e3c95e8bca3f732a49600a350217b8988afa8ce147cf61ea5e"} Jul 13 15:05:04.503707 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:04.503585 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:04.527416 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:04.527362 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" podStartSLOduration=1.5273460220000001 podStartE2EDuration="1.527346022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:05:04.524926623 +0000 UTC m=+1142.178893076" watchObservedRunningTime="2026-07-13 15:05:04.527346022 +0000 UTC m=+1142.181312457" Jul 13 15:05:15.510109 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:15.510069 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qhs4s" Jul 13 15:05:23.114214 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:23.114165 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:05:34.501377 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:34.501295 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:05:45.802128 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:45.802089 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:05:54.691905 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:05:54.691867 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:06:02.867257 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:02.867215 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:06:02.871963 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:02.871921 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:06:03.996365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:03.996334 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:06:13.906342 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:13.906303 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:06:24.302672 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:24.302629 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:06:34.845884 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:34.845839 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:06:44.896057 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:44.896019 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:06:55.207696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:06:55.204422 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:07:08.815316 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:07:08.815277 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:07:16.302629 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:07:16.302591 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:07:35.513222 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:07:35.513186 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:07:44.932585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:07:44.932554 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:07:54.719278 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:07:54.719241 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:08:04.102451 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:08:04.102414 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:08:11.394287 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:08:11.394237 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:08:27.406499 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:08:27.406401 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:08:37.502130 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:08:37.502095 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:08:47.149783 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:08:47.149742 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:08:56.824608 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:08:56.824567 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:09:06.200024 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:09:06.199989 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:09:15.598206 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:09:15.598169 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:09:28.907283 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:09:28.907240 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:09:38.011248 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:09:38.011201 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:09:47.502861 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:09:47.502820 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:09:57.200344 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:09:57.200258 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:10:09.904075 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:10:09.904040 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:10:17.410002 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:10:17.409965 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:10:25.996885 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:10:25.996843 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:10:40.500596 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:10:40.500560 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:10:51.901419 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:10:51.901387 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:11:02.899328 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:02.899219 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:11:02.904927 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:02.904905 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:11:04.612211 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:04.612180 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:11:34.311172 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:34.311077 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:11:43.002370 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:43.002332 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:11:51.808425 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:51.808387 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:11:55.228848 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.228782 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp"] Jul 13 15:11:55.233056 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.233033 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.238358 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.238327 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c4b04a32-openshift-default-dockercfg-vbnmw\"" Jul 13 15:11:55.278605 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.278570 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp"] Jul 13 15:11:55.369657 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369622 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spq2g\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-kube-api-access-spq2g\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-credential-socket\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369702 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-certs\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369748 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istiod-ca-cert\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369811 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-token\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369982 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-envoy\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369982 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-podinfo\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369982 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369944 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-data\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.369982 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.369962 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-socket\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471066 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.470969 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-token\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471066 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-envoy\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471364 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-podinfo\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471364 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-data\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471364 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-socket\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471364 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spq2g\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-kube-api-access-spq2g\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471364 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-credential-socket\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471364 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471336 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-certs\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471701 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istiod-ca-cert\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471701 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471608 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-data\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471701 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471657 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-socket\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.471896 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471872 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-certs\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.472017 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471997 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istiod-ca-cert\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.472074 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.471999 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-credential-socket\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.473438 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.473421 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-envoy\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.473531 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.473483 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-podinfo\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.482458 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.482433 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spq2g\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-kube-api-access-spq2g\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.482689 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.482668 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-token\") pod \"e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.545397 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.545358 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:55.690227 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.690199 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp"] Jul 13 15:11:55.691935 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:11:55.691902 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce0d7b1_f5b1_4e4b_a31e_329b38bed7fc.slice/crio-e9a468ebd82aa510f776e6494dab5c908ea916a5f2f6ff3782c8cf8e5a5137e2 WatchSource:0}: Error finding container e9a468ebd82aa510f776e6494dab5c908ea916a5f2f6ff3782c8cf8e5a5137e2: Status 404 returned error can't find the container with id e9a468ebd82aa510f776e6494dab5c908ea916a5f2f6ff3782c8cf8e5a5137e2 Jul 13 15:11:55.694018 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.693975 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:11:55.694092 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.694074 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:11:55.694131 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:55.694120 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:11:56.671617 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:56.671578 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" event={"ID":"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc","Type":"ContainerStarted","Data":"e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0"} Jul 13 15:11:56.672020 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:56.671642 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" event={"ID":"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc","Type":"ContainerStarted","Data":"e9a468ebd82aa510f776e6494dab5c908ea916a5f2f6ff3782c8cf8e5a5137e2"} Jul 13 15:11:56.702589 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:56.702540 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" podStartSLOduration=1.7025235890000001 podStartE2EDuration="1.702523589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:11:56.700396156 +0000 UTC m=+1554.354362614" watchObservedRunningTime="2026-07-13 15:11:56.702523589 +0000 UTC m=+1554.356490042" Jul 13 15:11:57.546423 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:57.546384 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:57.551188 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:57.551132 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:57.678200 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:57.678169 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:11:57.679105 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:11:57.679086 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:12:00.509395 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:00.509358 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:12:01.869119 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:01.869078 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp"] Jul 13 15:12:01.869568 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:01.869299 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" podUID="4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" containerName="istio-proxy" containerID="cri-o://e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0" gracePeriod=30 Jul 13 15:12:08.542529 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.542498 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:12:08.610357 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610272 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-socket\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610357 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610335 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-certs\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610551 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610362 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-envoy\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610551 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610424 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istiod-ca-cert\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610551 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610463 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-credential-socket\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610611 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:08.610696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610632 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:08.610696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610651 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spq2g\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-kube-api-access-spq2g\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610854 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610693 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-data\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610854 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610729 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-podinfo\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.610854 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610737 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:08.610854 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610761 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-token\") pod \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\" (UID: \"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc\") " Jul 13 15:12:08.611042 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.610845 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:12:08.611097 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.611078 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.611172 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.611096 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.611172 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.611111 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.611172 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.611126 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.611508 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.611481 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:08.612794 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.612763 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-kube-api-access-spq2g" (OuterVolumeSpecName: "kube-api-access-spq2g") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "kube-api-access-spq2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:12:08.612914 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.612844 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:12:08.613003 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.612986 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:08.613059 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.612982 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" (UID: "4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:12:08.712420 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.712380 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spq2g\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-kube-api-access-spq2g\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.712420 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.712413 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.712420 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.712425 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.712704 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.712436 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.712704 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.712448 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:08.736974 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.736939 2571 generic.go:356] "Generic (PLEG): container finished" podID="4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" containerID="e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0" exitCode=0 Jul 13 15:12:08.737178 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.736983 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" Jul 13 15:12:08.737178 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.737007 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" event={"ID":"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc","Type":"ContainerDied","Data":"e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0"} Jul 13 15:12:08.737178 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.737057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp" event={"ID":"4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc","Type":"ContainerDied","Data":"e9a468ebd82aa510f776e6494dab5c908ea916a5f2f6ff3782c8cf8e5a5137e2"} Jul 13 15:12:08.737178 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.737084 2571 scope.go:117] "RemoveContainer" containerID="e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0" Jul 13 15:12:08.747212 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.747186 2571 scope.go:117] "RemoveContainer" containerID="e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0" Jul 13 15:12:08.747520 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:12:08.747495 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0\": container with ID starting with e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0 not found: ID does not exist" containerID="e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0" Jul 13 15:12:08.747620 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.747525 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0"} err="failed to get container status \"e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0\": rpc error: code = NotFound desc = could not find container \"e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0\": container with ID starting with e18df314fc04ae7c3efc1a1cf5586502768e677e69418516920ea2ef82584cd0 not found: ID does not exist" Jul 13 15:12:08.765787 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.765760 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp"] Jul 13 15:12:08.770554 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.770530 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c4b04a32-openshift-default-68479d9b5-j8dzp"] Jul 13 15:12:08.874732 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:08.874630 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" path="/var/lib/kubelet/pods/4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc/volumes" Jul 13 15:12:09.036717 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.036668 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9"] Jul 13 15:12:09.037346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.037319 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" containerName="istio-proxy" Jul 13 15:12:09.037346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.037343 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" containerName="istio-proxy" Jul 13 15:12:09.037490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.037445 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce0d7b1-f5b1-4e4b-a31e-329b38bed7fc" containerName="istio-proxy" Jul 13 15:12:09.042411 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.042388 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.047566 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.047547 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a9109dfb-openshift-default-dockercfg-22sjt\"" Jul 13 15:12:09.056038 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.056015 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9"] Jul 13 15:12:09.117082 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117044 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istiod-ca-cert\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117277 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117087 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-certs\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117277 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117218 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-data\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117277 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117267 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-credential-socket\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117396 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117298 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-podinfo\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117396 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117325 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdtw\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-kube-api-access-pzdtw\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117396 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117369 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-socket\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117446 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-token\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.117490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.117470 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-envoy\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218238 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218107 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdtw\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-kube-api-access-pzdtw\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218238 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-socket\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-token\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-envoy\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218346 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istiod-ca-cert\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-certs\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218438 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-data\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218490 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-credential-socket\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218796 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218503 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-podinfo\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218796 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218646 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-socket\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218796 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218765 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-certs\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218963 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218850 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-data\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.218963 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.218869 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-credential-socket\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.219102 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.219080 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istiod-ca-cert\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.220603 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.220584 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-envoy\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.220887 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.220869 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-podinfo\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.232231 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.232203 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-token\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.233950 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.233927 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdtw\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-kube-api-access-pzdtw\") pod \"e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.353777 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.353738 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:09.511037 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.510956 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9"] Jul 13 15:12:09.526535 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:12:09.526505 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41d73fc_7736_4cea_ab74_bbc03e4c5552.slice/crio-e0d13045a971fbed2d57ced7ac2e83cfe93e7748a704e719e32316fa14c81a9f WatchSource:0}: Error finding container e0d13045a971fbed2d57ced7ac2e83cfe93e7748a704e719e32316fa14c81a9f: Status 404 returned error can't find the container with id e0d13045a971fbed2d57ced7ac2e83cfe93e7748a704e719e32316fa14c81a9f Jul 13 15:12:09.528791 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.528755 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:12:09.528875 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.528827 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:12:09.528912 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.528873 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:12:09.748943 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.748863 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" event={"ID":"d41d73fc-7736-4cea-ab74-bbc03e4c5552","Type":"ContainerStarted","Data":"71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94"} Jul 13 15:12:09.748943 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.748935 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" event={"ID":"d41d73fc-7736-4cea-ab74-bbc03e4c5552","Type":"ContainerStarted","Data":"e0d13045a971fbed2d57ced7ac2e83cfe93e7748a704e719e32316fa14c81a9f"} Jul 13 15:12:09.789900 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:09.789833 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" podStartSLOduration=0.789816353 podStartE2EDuration="789.816353ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:12:09.787718869 +0000 UTC m=+1567.441685322" watchObservedRunningTime="2026-07-13 15:12:09.789816353 +0000 UTC m=+1567.443782808" Jul 13 15:12:10.353959 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:10.353922 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:10.359484 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:10.359457 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:10.755778 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:10.755696 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:10.756758 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:10.756738 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:13.303512 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:13.303473 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:12:21.643946 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.643908 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9"] Jul 13 15:12:21.644471 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.644113 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" podUID="d41d73fc-7736-4cea-ab74-bbc03e4c5552" containerName="istio-proxy" containerID="cri-o://71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94" gracePeriod=30 Jul 13 15:12:21.792860 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.792833 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:21.810253 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.810223 2571 generic.go:356] "Generic (PLEG): container finished" podID="d41d73fc-7736-4cea-ab74-bbc03e4c5552" containerID="71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94" exitCode=0 Jul 13 15:12:21.810432 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.810264 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" Jul 13 15:12:21.810432 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.810315 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" event={"ID":"d41d73fc-7736-4cea-ab74-bbc03e4c5552","Type":"ContainerDied","Data":"71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94"} Jul 13 15:12:21.810432 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.810389 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9" event={"ID":"d41d73fc-7736-4cea-ab74-bbc03e4c5552","Type":"ContainerDied","Data":"e0d13045a971fbed2d57ced7ac2e83cfe93e7748a704e719e32316fa14c81a9f"} Jul 13 15:12:21.810597 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.810435 2571 scope.go:117] "RemoveContainer" containerID="71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94" Jul 13 15:12:21.819607 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.819563 2571 scope.go:117] "RemoveContainer" containerID="71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94" Jul 13 15:12:21.819900 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:12:21.819872 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94\": container with ID starting with 71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94 not found: ID does not exist" containerID="71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94" Jul 13 15:12:21.820019 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.819909 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94"} err="failed to get container status \"71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94\": rpc error: code = NotFound desc = could not find container \"71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94\": container with ID starting with 71c3a5bb968db84505f4308a5ffcb1238654b020e0e920dac3075682fccddc94 not found: ID does not exist" Jul 13 15:12:21.850610 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850580 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-token\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.850749 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850620 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istiod-ca-cert\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.850749 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850646 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-credential-socket\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.850749 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850675 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-podinfo\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.850749 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850698 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-certs\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.850749 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850717 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-data\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.851026 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850780 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzdtw\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-kube-api-access-pzdtw\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.851026 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850904 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-envoy\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.851026 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850939 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-socket\") pod \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\" (UID: \"d41d73fc-7736-4cea-ab74-bbc03e4c5552\") " Jul 13 15:12:21.851026 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.850987 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:21.851323 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.851304 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.851634 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.851612 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-data" (OuterVolumeSpecName: "istio-data") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:21.851736 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.851636 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:12:21.852066 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.852025 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:21.852261 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.852239 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:21.853457 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.853435 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:12:21.853541 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.853522 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-kube-api-access-pzdtw" (OuterVolumeSpecName: "kube-api-access-pzdtw") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "kube-api-access-pzdtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:12:21.853821 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.853801 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-token" (OuterVolumeSpecName: "istio-token") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:12:21.854020 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.853996 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d41d73fc-7736-4cea-ab74-bbc03e4c5552" (UID: "d41d73fc-7736-4cea-ab74-bbc03e4c5552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952308 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952337 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzdtw\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-kube-api-access-pzdtw\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952347 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952355 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952363 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952370 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952377 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41d73fc-7736-4cea-ab74-bbc03e4c5552-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:21.952400 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:21.952384 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41d73fc-7736-4cea-ab74-bbc03e4c5552-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:22.137956 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:22.137922 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9"] Jul 13 15:12:22.142694 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:22.142667 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a9109dfb-openshift-default-556c9c8967-89tz9"] Jul 13 15:12:22.871638 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:22.871600 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d41d73fc-7736-4cea-ab74-bbc03e4c5552" path="/var/lib/kubelet/pods/d41d73fc-7736-4cea-ab74-bbc03e4c5552/volumes" Jul 13 15:12:24.893351 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:24.893314 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:12:28.050014 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.049980 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb"] Jul 13 15:12:28.050636 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.050613 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d41d73fc-7736-4cea-ab74-bbc03e4c5552" containerName="istio-proxy" Jul 13 15:12:28.050636 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.050638 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d41d73fc-7736-4cea-ab74-bbc03e4c5552" containerName="istio-proxy" Jul 13 15:12:28.050784 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.050730 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d41d73fc-7736-4cea-ab74-bbc03e4c5552" containerName="istio-proxy" Jul 13 15:12:28.055260 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.055237 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.058494 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.058461 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-851a047e-openshift-default-dockercfg-qjqd4\"" Jul 13 15:12:28.067186 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.067145 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb"] Jul 13 15:12:28.210785 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.210742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070196f6-4ea5-4406-b48e-fcb0d015c905-istiod-ca-cert\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.210999 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.210791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-data\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.210999 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.210874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-certs\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.210999 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.210925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-token\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.210999 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.210948 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75hd7\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-kube-api-access-75hd7\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.210999 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.210997 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-envoy\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.211293 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.211022 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-socket\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.211293 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.211056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-credential-socket\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.211293 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.211099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-podinfo\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.311849 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.311759 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-token\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.311849 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.311796 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75hd7\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-kube-api-access-75hd7\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.311849 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.311838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-envoy\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.311857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-socket\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.311879 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-credential-socket\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.311905 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-podinfo\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312126 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070196f6-4ea5-4406-b48e-fcb0d015c905-istiod-ca-cert\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312424 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312190 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-data\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312424 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312279 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-certs\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312424 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312401 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-credential-socket\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312560 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312432 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-socket\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312621 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312595 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-data\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312663 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312617 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-certs\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.312780 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.312764 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070196f6-4ea5-4406-b48e-fcb0d015c905-istiod-ca-cert\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.314273 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.314253 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-envoy\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.314434 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.314395 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-podinfo\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.320819 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.320792 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-token\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.321202 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.321181 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75hd7\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-kube-api-access-75hd7\") pod \"e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.367549 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.367519 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:28.511799 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.511766 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb"] Jul 13 15:12:28.513883 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:12:28.513855 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070196f6_4ea5_4406_b48e_fcb0d015c905.slice/crio-fd1b110e6d897dc18a65ca75afeae11eb030344e47cd1e7abe30bdb3ccabe2d1 WatchSource:0}: Error finding container fd1b110e6d897dc18a65ca75afeae11eb030344e47cd1e7abe30bdb3ccabe2d1: Status 404 returned error can't find the container with id fd1b110e6d897dc18a65ca75afeae11eb030344e47cd1e7abe30bdb3ccabe2d1 Jul 13 15:12:28.516134 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.516098 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:12:28.516228 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.516210 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:12:28.516268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.516251 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:12:28.849133 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.849042 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" event={"ID":"070196f6-4ea5-4406-b48e-fcb0d015c905","Type":"ContainerStarted","Data":"812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d"} Jul 13 15:12:28.849133 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.849097 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" event={"ID":"070196f6-4ea5-4406-b48e-fcb0d015c905","Type":"ContainerStarted","Data":"fd1b110e6d897dc18a65ca75afeae11eb030344e47cd1e7abe30bdb3ccabe2d1"} Jul 13 15:12:28.873277 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:28.873202 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" podStartSLOduration=0.873187318 podStartE2EDuration="873.187318ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:12:28.871567398 +0000 UTC m=+1586.525533876" watchObservedRunningTime="2026-07-13 15:12:28.873187318 +0000 UTC m=+1586.527153859" Jul 13 15:12:29.368607 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:29.368578 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:29.373900 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:29.373871 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:29.855086 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:29.855054 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:29.856162 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:29.856131 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:33.210935 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:33.210899 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:12:33.983216 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:33.983180 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb"] Jul 13 15:12:33.983411 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:33.983368 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" podUID="070196f6-4ea5-4406-b48e-fcb0d015c905" containerName="istio-proxy" containerID="cri-o://812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d" gracePeriod=30 Jul 13 15:12:39.247964 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.247898 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:39.426138 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426089 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-credential-socket\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426138 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426179 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75hd7\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-kube-api-access-75hd7\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426448 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426225 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070196f6-4ea5-4406-b48e-fcb0d015c905-istiod-ca-cert\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426448 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426271 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-podinfo\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426448 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426318 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-data\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426448 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426338 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-certs\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426448 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426357 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-socket\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426448 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426396 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-envoy\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426448 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426445 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-token\") pod \"070196f6-4ea5-4406-b48e-fcb0d015c905\" (UID: \"070196f6-4ea5-4406-b48e-fcb0d015c905\") " Jul 13 15:12:39.426813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426477 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:39.426813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426672 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:39.426813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426708 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:39.426813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426749 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.426813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426767 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.427090 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.426982 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070196f6-4ea5-4406-b48e-fcb0d015c905-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:12:39.427236 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.427209 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-data" (OuterVolumeSpecName: "istio-data") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:39.428470 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.428447 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-kube-api-access-75hd7" (OuterVolumeSpecName: "kube-api-access-75hd7") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "kube-api-access-75hd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:12:39.428604 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.428567 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:12:39.428822 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.428797 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:12:39.428920 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.428862 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-token" (OuterVolumeSpecName: "istio-token") pod "070196f6-4ea5-4406-b48e-fcb0d015c905" (UID: "070196f6-4ea5-4406-b48e-fcb0d015c905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:12:39.527346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.527258 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070196f6-4ea5-4406-b48e-fcb0d015c905-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.527346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.527289 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.527346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.527298 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.527346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.527306 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.527346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.527313 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.527346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.527321 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.527346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.527329 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75hd7\" (UniqueName: \"kubernetes.io/projected/070196f6-4ea5-4406-b48e-fcb0d015c905-kube-api-access-75hd7\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:12:39.908321 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.908284 2571 generic.go:356] "Generic (PLEG): container finished" podID="070196f6-4ea5-4406-b48e-fcb0d015c905" containerID="812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d" exitCode=0 Jul 13 15:12:39.908506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.908341 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" Jul 13 15:12:39.908506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.908346 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" event={"ID":"070196f6-4ea5-4406-b48e-fcb0d015c905","Type":"ContainerDied","Data":"812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d"} Jul 13 15:12:39.908506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.908397 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb" event={"ID":"070196f6-4ea5-4406-b48e-fcb0d015c905","Type":"ContainerDied","Data":"fd1b110e6d897dc18a65ca75afeae11eb030344e47cd1e7abe30bdb3ccabe2d1"} Jul 13 15:12:39.908506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.908421 2571 scope.go:117] "RemoveContainer" containerID="812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d" Jul 13 15:12:39.919547 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.919530 2571 scope.go:117] "RemoveContainer" containerID="812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d" Jul 13 15:12:39.919799 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:12:39.919783 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d\": container with ID starting with 812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d not found: ID does not exist" containerID="812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d" Jul 13 15:12:39.919847 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.919807 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d"} err="failed to get container status \"812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d\": rpc error: code = NotFound desc = could not find container \"812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d\": container with ID starting with 812f831d2d59ce57410e23b313209ebeaafa9c5d7cf0fb6948be3131cc964c4d not found: ID does not exist" Jul 13 15:12:39.935301 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.935272 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb"] Jul 13 15:12:39.940359 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:39.940330 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-851a047e-openshift-default-5c4b4dcfb7-zlnlb"] Jul 13 15:12:40.871028 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:40.870989 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="070196f6-4ea5-4406-b48e-fcb0d015c905" path="/var/lib/kubelet/pods/070196f6-4ea5-4406-b48e-fcb0d015c905/volumes" Jul 13 15:12:41.735776 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:12:41.735740 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:14:05.211914 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:05.211879 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:14:12.912000 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:12.911964 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:14:19.664311 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.664273 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq"] Jul 13 15:14:19.664925 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.664885 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070196f6-4ea5-4406-b48e-fcb0d015c905" containerName="istio-proxy" Jul 13 15:14:19.664925 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.664918 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="070196f6-4ea5-4406-b48e-fcb0d015c905" containerName="istio-proxy" Jul 13 15:14:19.665120 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.665055 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="070196f6-4ea5-4406-b48e-fcb0d015c905" containerName="istio-proxy" Jul 13 15:14:19.668238 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.668220 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.674263 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.674236 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0a904fe5-openshift-default-dockercfg-9lvbx\"" Jul 13 15:14:19.705958 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.705928 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq"] Jul 13 15:14:19.804728 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.804696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af612504-af71-4c63-87e4-bbe1a163f9c7-istiod-ca-cert\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.804916 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.804743 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7bnz\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-kube-api-access-l7bnz\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.804916 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.804821 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-credential-socket\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.804916 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.804877 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-envoy\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.805042 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.804921 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-data\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.805042 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.804942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-certs\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.805042 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.804964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-socket\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.805042 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.805001 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-token\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.805231 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.805058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-podinfo\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906449 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-credential-socket\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906449 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906450 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-envoy\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906663 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-data\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906663 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-certs\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906663 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906645 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-socket\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-token\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906693 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-podinfo\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906760 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af612504-af71-4c63-87e4-bbe1a163f9c7-istiod-ca-cert\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.906826 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.906812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bnz\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-kube-api-access-l7bnz\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.907087 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.907061 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-credential-socket\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.907254 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.907106 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-certs\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.907322 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.907165 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-socket\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.907322 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.907269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-data\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.907542 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.907525 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af612504-af71-4c63-87e4-bbe1a163f9c7-istiod-ca-cert\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.908744 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.908722 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-envoy\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.909206 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.909187 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-podinfo\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.920692 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.920625 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-token\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.920837 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.920780 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7bnz\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-kube-api-access-l7bnz\") pod \"e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:19.979506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:19.979476 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:20.133146 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.133118 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq"] Jul 13 15:14:20.135326 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:14:20.135289 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf612504_af71_4c63_87e4_bbe1a163f9c7.slice/crio-e2c05dfa4d92aff56a4b0e736e1452d48301f7cf42cc44c55f971bced1c9ecf7 WatchSource:0}: Error finding container e2c05dfa4d92aff56a4b0e736e1452d48301f7cf42cc44c55f971bced1c9ecf7: Status 404 returned error can't find the container with id e2c05dfa4d92aff56a4b0e736e1452d48301f7cf42cc44c55f971bced1c9ecf7 Jul 13 15:14:20.139183 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.138988 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:14:20.139183 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.139082 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:14:20.139183 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.139127 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:14:20.409928 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.409887 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:14:20.423009 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.422974 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" event={"ID":"af612504-af71-4c63-87e4-bbe1a163f9c7","Type":"ContainerStarted","Data":"aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21"} Jul 13 15:14:20.423209 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.423034 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" event={"ID":"af612504-af71-4c63-87e4-bbe1a163f9c7","Type":"ContainerStarted","Data":"e2c05dfa4d92aff56a4b0e736e1452d48301f7cf42cc44c55f971bced1c9ecf7"} Jul 13 15:14:20.446669 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.446553 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" podStartSLOduration=1.446530661 podStartE2EDuration="1.446530661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:14:20.44512294 +0000 UTC m=+1698.099089394" watchObservedRunningTime="2026-07-13 15:14:20.446530661 +0000 UTC m=+1698.100497116" Jul 13 15:14:20.979835 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:20.979801 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:21.985439 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:21.985405 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:22.432625 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:22.432592 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:22.433600 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:22.433582 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:28.313972 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:28.313891 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:14:36.508889 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:36.508802 2571 generic.go:356] "Generic (PLEG): container finished" podID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerID="aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21" exitCode=0 Jul 13 15:14:36.508889 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:36.508875 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" event={"ID":"af612504-af71-4c63-87e4-bbe1a163f9c7","Type":"ContainerDied","Data":"aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21"} Jul 13 15:14:36.509365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:36.509313 2571 scope.go:117] "RemoveContainer" containerID="aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21" Jul 13 15:14:36.510282 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:36.510253 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:14:36.510344 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:36.510322 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:14:36.510391 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:36.510352 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:14:37.433444 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:37.433408 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:37.517419 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:37.517381 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" event={"ID":"af612504-af71-4c63-87e4-bbe1a163f9c7","Type":"ContainerStarted","Data":"53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3"} Jul 13 15:14:37.980723 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:37.980686 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:37.981911 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:37.981883 2571 patch_prober.go:28] interesting pod/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 13 15:14:37.982033 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:37.981946 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 13 15:14:38.980129 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:38.980090 2571 patch_prober.go:28] interesting pod/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 13 15:14:38.980570 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:38.980183 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 13 15:14:39.980353 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:39.980315 2571 patch_prober.go:28] interesting pod/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 13 15:14:39.980837 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:39.980380 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 13 15:14:40.980530 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:40.980490 2571 patch_prober.go:28] interesting pod/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 13 15:14:40.980928 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:40.980550 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 13 15:14:41.925011 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:41.924971 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:14:41.984388 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:41.984358 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:41.984750 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:41.984627 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:41.985419 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:41.985397 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:43.179437 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:43.179402 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq"] Jul 13 15:14:43.549242 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:43.549194 2571 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" secret="" err="secret \"e2e-mt-0a904fe5-openshift-default-dockercfg-9lvbx\" not found" Jul 13 15:14:44.555074 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:44.555009 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" containerID="cri-o://53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3" gracePeriod=30 Jul 13 15:14:49.822345 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.822279 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:49.907973 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.907933 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-envoy\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.907973 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.907979 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-data\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908245 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908001 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af612504-af71-4c63-87e4-bbe1a163f9c7-istiod-ca-cert\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908245 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908018 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-token\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908245 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908067 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7bnz\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-kube-api-access-l7bnz\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908245 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908103 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-credential-socket\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908245 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908124 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-certs\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908245 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908191 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-podinfo\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908614 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908284 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-socket\") pod \"af612504-af71-4c63-87e4-bbe1a163f9c7\" (UID: \"af612504-af71-4c63-87e4-bbe1a163f9c7\") " Jul 13 15:14:49.908614 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908433 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:14:49.908614 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908442 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af612504-af71-4c63-87e4-bbe1a163f9c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:14:49.908777 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908638 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:14:49.908777 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908711 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:14:49.908893 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908787 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af612504-af71-4c63-87e4-bbe1a163f9c7-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:49.908893 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908817 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:49.908893 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908832 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:49.909046 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.908967 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:14:49.910362 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.910333 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-kube-api-access-l7bnz" (OuterVolumeSpecName: "kube-api-access-l7bnz") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "kube-api-access-l7bnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:14:49.910561 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.910544 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:14:49.910659 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.910641 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:14:49.910705 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:49.910658 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "af612504-af71-4c63-87e4-bbe1a163f9c7" (UID: "af612504-af71-4c63-87e4-bbe1a163f9c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:14:50.009413 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.009374 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:50.009413 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.009401 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:50.009413 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.009410 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:50.009413 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.009419 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:50.009413 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.009427 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:50.009743 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.009435 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7bnz\" (UniqueName: \"kubernetes.io/projected/af612504-af71-4c63-87e4-bbe1a163f9c7-kube-api-access-l7bnz\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:14:50.587084 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.587047 2571 generic.go:356] "Generic (PLEG): container finished" podID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerID="53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3" exitCode=0 Jul 13 15:14:50.587296 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.587097 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" Jul 13 15:14:50.587296 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.587097 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" event={"ID":"af612504-af71-4c63-87e4-bbe1a163f9c7","Type":"ContainerDied","Data":"53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3"} Jul 13 15:14:50.587296 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.587220 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq" event={"ID":"af612504-af71-4c63-87e4-bbe1a163f9c7","Type":"ContainerDied","Data":"e2c05dfa4d92aff56a4b0e736e1452d48301f7cf42cc44c55f971bced1c9ecf7"} Jul 13 15:14:50.587296 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.587249 2571 scope.go:117] "RemoveContainer" containerID="53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3" Jul 13 15:14:50.596684 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.596664 2571 scope.go:117] "RemoveContainer" containerID="aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21" Jul 13 15:14:50.604479 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.604459 2571 scope.go:117] "RemoveContainer" containerID="53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3" Jul 13 15:14:50.604807 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:14:50.604779 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3\": container with ID starting with 53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3 not found: ID does not exist" containerID="53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3" Jul 13 15:14:50.604908 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.604818 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3"} err="failed to get container status \"53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3\": rpc error: code = NotFound desc = could not find container \"53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3\": container with ID starting with 53f96a7b57089ccaf4b9458d3322db6e9963168d2351eb2feb5e8109642bb1b3 not found: ID does not exist" Jul 13 15:14:50.604908 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.604845 2571 scope.go:117] "RemoveContainer" containerID="aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21" Jul 13 15:14:50.605143 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:14:50.605122 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21\": container with ID starting with aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21 not found: ID does not exist" containerID="aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21" Jul 13 15:14:50.605218 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.605164 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21"} err="failed to get container status \"aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21\": rpc error: code = NotFound desc = could not find container \"aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21\": container with ID starting with aae4f7ad8128d43d22f120d6332f56b9a98cb7a354f1c38e710d0bc9af2f4d21 not found: ID does not exist" Jul 13 15:14:50.611356 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.611333 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq"] Jul 13 15:14:50.619038 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.619015 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0a904fe5-openshift-default-55fdc84ff-4slsq"] Jul 13 15:14:50.871084 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:14:50.870988 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" path="/var/lib/kubelet/pods/af612504-af71-4c63-87e4-bbe1a163f9c7/volumes" Jul 13 15:15:00.165588 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.165546 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29732595-d5wxg"] Jul 13 15:15:00.166195 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.166177 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" Jul 13 15:15:00.166278 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.166198 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" Jul 13 15:15:00.166339 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.166329 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" Jul 13 15:15:00.166399 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.166345 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" Jul 13 15:15:00.171441 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.171410 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" Jul 13 15:15:00.181661 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.181636 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9vf7z\"" Jul 13 15:15:00.189938 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.189894 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29732595-d5wxg"] Jul 13 15:15:00.309601 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.309560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4q8\" (UniqueName: \"kubernetes.io/projected/51d85393-a96c-4ac1-9013-38ce1a397313-kube-api-access-vn4q8\") pod \"maas-api-key-cleanup-29732595-d5wxg\" (UID: \"51d85393-a96c-4ac1-9013-38ce1a397313\") " pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" Jul 13 15:15:00.410482 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.410432 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4q8\" (UniqueName: \"kubernetes.io/projected/51d85393-a96c-4ac1-9013-38ce1a397313-kube-api-access-vn4q8\") pod \"maas-api-key-cleanup-29732595-d5wxg\" (UID: \"51d85393-a96c-4ac1-9013-38ce1a397313\") " pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" Jul 13 15:15:00.431684 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.431611 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4q8\" (UniqueName: \"kubernetes.io/projected/51d85393-a96c-4ac1-9013-38ce1a397313-kube-api-access-vn4q8\") pod \"maas-api-key-cleanup-29732595-d5wxg\" (UID: \"51d85393-a96c-4ac1-9013-38ce1a397313\") " pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" Jul 13 15:15:00.489974 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.489941 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" Jul 13 15:15:00.639053 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:00.639023 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29732595-d5wxg"] Jul 13 15:15:00.640368 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:15:00.640341 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d85393_a96c_4ac1_9013_38ce1a397313.slice/crio-62ab799e3b99f53b94173eb8dbb21230c0b8bfee4a629f47e55514a6cf8e59d7 WatchSource:0}: Error finding container 62ab799e3b99f53b94173eb8dbb21230c0b8bfee4a629f47e55514a6cf8e59d7: Status 404 returned error can't find the container with id 62ab799e3b99f53b94173eb8dbb21230c0b8bfee4a629f47e55514a6cf8e59d7 Jul 13 15:15:01.643555 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:01.643521 2571 generic.go:356] "Generic (PLEG): container finished" podID="51d85393-a96c-4ac1-9013-38ce1a397313" containerID="dc80c710ecdb645697b14b902bbc585a740c7a820237c0dfbae0894d2cd22064" exitCode=0 Jul 13 15:15:01.644035 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:01.643552 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" event={"ID":"51d85393-a96c-4ac1-9013-38ce1a397313","Type":"ContainerDied","Data":"dc80c710ecdb645697b14b902bbc585a740c7a820237c0dfbae0894d2cd22064"} Jul 13 15:15:01.644035 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:01.643618 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" event={"ID":"51d85393-a96c-4ac1-9013-38ce1a397313","Type":"ContainerStarted","Data":"62ab799e3b99f53b94173eb8dbb21230c0b8bfee4a629f47e55514a6cf8e59d7"} Jul 13 15:15:02.783098 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:02.783074 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" Jul 13 15:15:02.931440 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:02.931355 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn4q8\" (UniqueName: \"kubernetes.io/projected/51d85393-a96c-4ac1-9013-38ce1a397313-kube-api-access-vn4q8\") pod \"51d85393-a96c-4ac1-9013-38ce1a397313\" (UID: \"51d85393-a96c-4ac1-9013-38ce1a397313\") " Jul 13 15:15:02.933321 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:02.933289 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d85393-a96c-4ac1-9013-38ce1a397313-kube-api-access-vn4q8" (OuterVolumeSpecName: "kube-api-access-vn4q8") pod "51d85393-a96c-4ac1-9013-38ce1a397313" (UID: "51d85393-a96c-4ac1-9013-38ce1a397313"). InnerVolumeSpecName "kube-api-access-vn4q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:15:03.032901 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:03.032873 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4q8\" (UniqueName: \"kubernetes.io/projected/51d85393-a96c-4ac1-9013-38ce1a397313-kube-api-access-vn4q8\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:15:03.656912 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:03.656881 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" Jul 13 15:15:03.656912 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:03.656890 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29732595-d5wxg" event={"ID":"51d85393-a96c-4ac1-9013-38ce1a397313","Type":"ContainerDied","Data":"62ab799e3b99f53b94173eb8dbb21230c0b8bfee4a629f47e55514a6cf8e59d7"} Jul 13 15:15:03.657119 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:03.656944 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62ab799e3b99f53b94173eb8dbb21230c0b8bfee4a629f47e55514a6cf8e59d7" Jul 13 15:15:04.119205 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.119140 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24"] Jul 13 15:15:04.119777 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.119755 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d85393-a96c-4ac1-9013-38ce1a397313" containerName="cleanup" Jul 13 15:15:04.119777 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.119778 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d85393-a96c-4ac1-9013-38ce1a397313" containerName="cleanup" Jul 13 15:15:04.119915 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.119806 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" Jul 13 15:15:04.119915 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.119815 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="af612504-af71-4c63-87e4-bbe1a163f9c7" containerName="istio-proxy" Jul 13 15:15:04.119915 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.119906 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d85393-a96c-4ac1-9013-38ce1a397313" containerName="cleanup" Jul 13 15:15:04.128838 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.128808 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.133126 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.133101 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-59ce9939-openshift-default-dockercfg-n658b\"" Jul 13 15:15:04.145912 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.145877 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24"] Jul 13 15:15:04.245327 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-socket\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245327 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245322 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-certs\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245580 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-envoy\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245580 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245415 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-podinfo\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245580 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245454 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-data\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245580 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glj4t\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-kube-api-access-glj4t\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245580 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-credential-socket\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245860 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8766c9-7f11-4f69-b88b-7daed84fb170-istiod-ca-cert\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.245860 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.245647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-token\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347175 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347126 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-token\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-socket\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347251 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-certs\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347268 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-envoy\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347305 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-podinfo\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347328 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-data\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347643 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glj4t\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-kube-api-access-glj4t\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347643 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-credential-socket\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347643 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8766c9-7f11-4f69-b88b-7daed84fb170-istiod-ca-cert\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347800 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347713 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-certs\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347800 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347734 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-socket\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.347909 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.347818 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-data\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.348144 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.348110 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8766c9-7f11-4f69-b88b-7daed84fb170-istiod-ca-cert\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.348342 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.348317 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-credential-socket\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.350054 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.350028 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-envoy\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.350246 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.350230 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-podinfo\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.361611 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.361583 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glj4t\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-kube-api-access-glj4t\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.364924 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.364900 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-token\") pod \"e2e-mt-59ce9939-openshift-default-74b695497f-x9f24\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.441983 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.441007 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:04.600299 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.600265 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24"] Jul 13 15:15:04.602547 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:15:04.602520 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8766c9_7f11_4f69_b88b_7daed84fb170.slice/crio-345120d3bff98ffa8b953309ce5111bdf93bf6c68bc2e45af8cf5dfabccee8f3 WatchSource:0}: Error finding container 345120d3bff98ffa8b953309ce5111bdf93bf6c68bc2e45af8cf5dfabccee8f3: Status 404 returned error can't find the container with id 345120d3bff98ffa8b953309ce5111bdf93bf6c68bc2e45af8cf5dfabccee8f3 Jul 13 15:15:04.605282 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.605247 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:15:04.605375 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.605325 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:15:04.605375 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.605359 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:15:04.664488 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:04.664460 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" event={"ID":"4c8766c9-7f11-4f69-b88b-7daed84fb170","Type":"ContainerStarted","Data":"345120d3bff98ffa8b953309ce5111bdf93bf6c68bc2e45af8cf5dfabccee8f3"} Jul 13 15:15:05.671582 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:05.671548 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" event={"ID":"4c8766c9-7f11-4f69-b88b-7daed84fb170","Type":"ContainerStarted","Data":"69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f"} Jul 13 15:15:05.701781 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:05.701724 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" podStartSLOduration=1.701708622 podStartE2EDuration="1.701708622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:15:05.69749399 +0000 UTC m=+1743.351460444" watchObservedRunningTime="2026-07-13 15:15:05.701708622 +0000 UTC m=+1743.355675075" Jul 13 15:15:06.441954 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:06.441918 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:06.446783 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:06.446759 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:06.676916 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:06.676880 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:06.677937 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:06.677913 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:12.943969 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:12.943935 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:15:21.678527 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.678492 2571 patch_prober.go:28] interesting pod/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 13 15:15:21.678975 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.678571 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 13 15:15:21.773180 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.773076 2571 generic.go:356] "Generic (PLEG): container finished" podID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerID="69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f" exitCode=0 Jul 13 15:15:21.773357 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.773168 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" event={"ID":"4c8766c9-7f11-4f69-b88b-7daed84fb170","Type":"ContainerDied","Data":"69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f"} Jul 13 15:15:21.773713 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.773695 2571 scope.go:117] "RemoveContainer" containerID="69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f" Jul 13 15:15:21.774935 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.774896 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:15:21.775007 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.774990 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:15:21.775045 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:21.775036 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:15:22.780571 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:22.780536 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" event={"ID":"4c8766c9-7f11-4f69-b88b-7daed84fb170","Type":"ContainerStarted","Data":"86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b"} Jul 13 15:15:23.441674 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:23.441642 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:23.442104 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:23.442069 2571 patch_prober.go:28] interesting pod/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 13 15:15:23.442249 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:23.442171 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 13 15:15:24.441769 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:24.441720 2571 patch_prober.go:28] interesting pod/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 13 15:15:24.442347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:24.441793 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 13 15:15:25.442503 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:25.442464 2571 patch_prober.go:28] interesting pod/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 13 15:15:25.443004 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:25.442533 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 13 15:15:26.442107 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:26.442061 2571 patch_prober.go:28] interesting pod/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 13 15:15:26.442342 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:26.442127 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 13 15:15:27.445841 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:27.445806 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:27.446344 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:27.446095 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:27.446761 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:27.446740 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:15:34.707809 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:34.707774 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:15:47.413596 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:15:47.413560 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:16:02.107423 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:02.107385 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:16:02.930208 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:02.930180 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:16:02.937407 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:02.937371 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:16:41.507115 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:41.507076 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:16:51.445841 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:51.445804 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24"] Jul 13 15:16:51.446346 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:51.446122 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" containerID="cri-o://86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b" gracePeriod=30 Jul 13 15:16:55.275733 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:55.275695 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:16:56.728147 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.728075 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:16:56.845956 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.845920 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-socket\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.845956 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.845963 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-token\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.845987 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-certs\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846031 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-envoy\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846085 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-data\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846180 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8766c9-7f11-4f69-b88b-7daed84fb170-istiod-ca-cert\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846209 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-credential-socket\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846235 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-podinfo\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846241 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:16:56.846268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846267 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glj4t\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-kube-api-access-glj4t\") pod \"4c8766c9-7f11-4f69-b88b-7daed84fb170\" (UID: \"4c8766c9-7f11-4f69-b88b-7daed84fb170\") " Jul 13 15:16:56.846700 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846424 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:16:56.846700 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846587 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8766c9-7f11-4f69-b88b-7daed84fb170-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:16:56.846817 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846699 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:16:56.846817 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846708 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.846817 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846735 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8766c9-7f11-4f69-b88b-7daed84fb170-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.846817 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.846752 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.847432 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.847389 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:16:56.848505 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.848474 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:16:56.848608 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.848592 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:16:56.848716 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.848687 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:16:56.848898 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.848878 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-kube-api-access-glj4t" (OuterVolumeSpecName: "kube-api-access-glj4t") pod "4c8766c9-7f11-4f69-b88b-7daed84fb170" (UID: "4c8766c9-7f11-4f69-b88b-7daed84fb170"). InnerVolumeSpecName "kube-api-access-glj4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:16:56.948044 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.948002 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.948044 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.948033 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.948044 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.948044 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glj4t\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-kube-api-access-glj4t\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.948044 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.948054 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.948442 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.948063 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:56.948442 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:56.948071 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8766c9-7f11-4f69-b88b-7daed84fb170-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:16:57.276596 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.276552 2571 generic.go:356] "Generic (PLEG): container finished" podID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerID="86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b" exitCode=0 Jul 13 15:16:57.276748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.276603 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" event={"ID":"4c8766c9-7f11-4f69-b88b-7daed84fb170","Type":"ContainerDied","Data":"86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b"} Jul 13 15:16:57.276748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.276648 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" event={"ID":"4c8766c9-7f11-4f69-b88b-7daed84fb170","Type":"ContainerDied","Data":"345120d3bff98ffa8b953309ce5111bdf93bf6c68bc2e45af8cf5dfabccee8f3"} Jul 13 15:16:57.276748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.276669 2571 scope.go:117] "RemoveContainer" containerID="86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b" Jul 13 15:16:57.276748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.276607 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24" Jul 13 15:16:57.288850 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.288824 2571 scope.go:117] "RemoveContainer" containerID="69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f" Jul 13 15:16:57.297512 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.297492 2571 scope.go:117] "RemoveContainer" containerID="86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b" Jul 13 15:16:57.297837 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:16:57.297810 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b\": container with ID starting with 86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b not found: ID does not exist" containerID="86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b" Jul 13 15:16:57.297890 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.297864 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b"} err="failed to get container status \"86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b\": rpc error: code = NotFound desc = could not find container \"86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b\": container with ID starting with 86f9d67c4130150eb39149dbd388e89655027e0e1c605e5dec0f3fbeb6e9d98b not found: ID does not exist" Jul 13 15:16:57.297890 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.297884 2571 scope.go:117] "RemoveContainer" containerID="69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f" Jul 13 15:16:57.298136 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:16:57.298112 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f\": container with ID starting with 69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f not found: ID does not exist" containerID="69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f" Jul 13 15:16:57.298205 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.298147 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f"} err="failed to get container status \"69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f\": rpc error: code = NotFound desc = could not find container \"69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f\": container with ID starting with 69420b48e243ea0e4f26fb713bdf3c0a741a0c6398420f6a605a7fb4afd33b4f not found: ID does not exist" Jul 13 15:16:57.310761 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.310739 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24"] Jul 13 15:16:57.317596 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:57.317574 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-59ce9939-openshift-default-74b695497f-x9f24"] Jul 13 15:16:58.869917 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:16:58.869882 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" path="/var/lib/kubelet/pods/4c8766c9-7f11-4f69-b88b-7daed84fb170/volumes" Jul 13 15:17:06.722900 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:06.722858 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:17:49.932023 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.931989 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms"] Jul 13 15:17:49.932506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.932420 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" Jul 13 15:17:49.932506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.932433 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" Jul 13 15:17:49.932506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.932442 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" Jul 13 15:17:49.932506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.932448 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" Jul 13 15:17:49.932506 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.932508 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" Jul 13 15:17:49.932671 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.932517 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8766c9-7f11-4f69-b88b-7daed84fb170" containerName="istio-proxy" Jul 13 15:17:49.935809 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.935785 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:49.939395 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.939373 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d5c92a-openshift-default-dockercfg-5g68h\"" Jul 13 15:17:49.950614 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:49.950589 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms"] Jul 13 15:17:50.033955 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.033921 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-socket\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034144 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.033960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-certs\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034144 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.033990 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istiod-ca-cert\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034144 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.034116 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-data\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034387 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.034230 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-podinfo\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034387 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.034282 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-credential-socket\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034387 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.034345 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-envoy\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034542 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.034385 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdjh\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-kube-api-access-vfdjh\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.034542 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.034424 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-token\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.134966 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.134928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-socket\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135199 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.134977 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-certs\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135199 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istiod-ca-cert\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135199 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-data\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135199 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-podinfo\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135199 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-credential-socket\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135513 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-envoy\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135513 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdjh\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-kube-api-access-vfdjh\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135513 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135326 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-token\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135513 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135403 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-socket\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.135813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.135788 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-certs\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.136039 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.136016 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-credential-socket\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.136306 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.136285 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-data\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.136456 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.136434 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istiod-ca-cert\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.137727 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.137696 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-envoy\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.138533 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.138510 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-podinfo\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.149611 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.149588 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdjh\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-kube-api-access-vfdjh\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.150962 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.150943 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-token\") pod \"e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.247678 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.247601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:50.408181 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.408144 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms"] Jul 13 15:17:50.410068 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:17:50.410043 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ceec98_6c45_4e4d_a3e9_c6f1c555adaf.slice/crio-f9f2d09100ec08265973aff10d60ddb42595593497fb99bbb08c10afeb7b6bc3 WatchSource:0}: Error finding container f9f2d09100ec08265973aff10d60ddb42595593497fb99bbb08c10afeb7b6bc3: Status 404 returned error can't find the container with id f9f2d09100ec08265973aff10d60ddb42595593497fb99bbb08c10afeb7b6bc3 Jul 13 15:17:50.412194 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.412148 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:17:50.412258 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.412232 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:17:50.412301 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.412262 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:17:50.548996 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.548966 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" event={"ID":"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf","Type":"ContainerStarted","Data":"78b30348a0ca9c34ed430bc887c55b71f19ef10fb2d19d3200f83d2104e9551f"} Jul 13 15:17:50.549213 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.549014 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" event={"ID":"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf","Type":"ContainerStarted","Data":"f9f2d09100ec08265973aff10d60ddb42595593497fb99bbb08c10afeb7b6bc3"} Jul 13 15:17:50.579946 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:50.579895 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" podStartSLOduration=1.5798774469999999 podStartE2EDuration="1.579877447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:17:50.575282496 +0000 UTC m=+1908.229248949" watchObservedRunningTime="2026-07-13 15:17:50.579877447 +0000 UTC m=+1908.233843902" Jul 13 15:17:51.248458 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:51.248423 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:51.253467 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:51.253442 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:51.554590 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:51.554558 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:51.555557 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:51.555538 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:17:56.919983 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:17:56.919947 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:18:05.638560 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:05.638504 2571 generic.go:356] "Generic (PLEG): container finished" podID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerID="78b30348a0ca9c34ed430bc887c55b71f19ef10fb2d19d3200f83d2104e9551f" exitCode=0 Jul 13 15:18:05.639077 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:05.638579 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" event={"ID":"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf","Type":"ContainerDied","Data":"78b30348a0ca9c34ed430bc887c55b71f19ef10fb2d19d3200f83d2104e9551f"} Jul 13 15:18:05.639671 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:05.639653 2571 scope.go:117] "RemoveContainer" containerID="78b30348a0ca9c34ed430bc887c55b71f19ef10fb2d19d3200f83d2104e9551f" Jul 13 15:18:05.640993 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:05.640936 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:18:05.641110 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:05.641053 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:18:05.641110 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:05.641104 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:18:06.554941 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:06.554891 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:18:06.650501 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:06.650456 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" event={"ID":"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf","Type":"ContainerStarted","Data":"2ae74628911f36e027d7814c7a99dbdbc77abcd868f978fa20937fb79c224c83"} Jul 13 15:18:07.249271 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:07.249232 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:18:07.249485 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:07.249407 2571 patch_prober.go:28] interesting pod/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 13 15:18:07.249560 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:07.249489 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 13 15:18:07.809654 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:07.809614 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:18:08.248919 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:08.248824 2571 patch_prober.go:28] interesting pod/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 13 15:18:08.249239 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:08.249199 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 13 15:18:09.248829 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:09.248788 2571 patch_prober.go:28] interesting pod/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 13 15:18:09.249840 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:09.249431 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 13 15:18:10.247995 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:10.247949 2571 patch_prober.go:28] interesting pod/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 13 15:18:10.248236 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:10.248022 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 13 15:18:11.252512 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:11.252478 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:18:11.252903 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:11.252756 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:18:11.253662 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:11.253640 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:18:16.901186 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:16.901133 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:18:27.706629 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:27.706590 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:18:39.102836 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:39.102801 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:18:54.397426 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:18:54.397386 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:19:32.990328 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:32.990283 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms"] Jul 13 15:19:32.990838 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:32.990550 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" containerID="cri-o://2ae74628911f36e027d7814c7a99dbdbc77abcd868f978fa20937fb79c224c83" gracePeriod=30 Jul 13 15:19:37.102304 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:37.102263 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:19:38.128318 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.128286 2571 generic.go:356] "Generic (PLEG): container finished" podID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerID="2ae74628911f36e027d7814c7a99dbdbc77abcd868f978fa20937fb79c224c83" exitCode=0 Jul 13 15:19:38.128798 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.128355 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" event={"ID":"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf","Type":"ContainerDied","Data":"2ae74628911f36e027d7814c7a99dbdbc77abcd868f978fa20937fb79c224c83"} Jul 13 15:19:38.128798 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.128419 2571 scope.go:117] "RemoveContainer" containerID="78b30348a0ca9c34ed430bc887c55b71f19ef10fb2d19d3200f83d2104e9551f" Jul 13 15:19:38.260549 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.260479 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:19:38.320338 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320310 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-data\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320338 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320343 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-podinfo\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320597 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320372 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-token\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320597 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320401 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-credential-socket\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320597 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320463 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-certs\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320597 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320487 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-envoy\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320597 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320534 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istiod-ca-cert\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320597 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320567 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-socket\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320931 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320613 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfdjh\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-kube-api-access-vfdjh\") pod \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\" (UID: \"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf\") " Jul 13 15:19:38.320931 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320778 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:19:38.320931 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320929 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.321114 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320884 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:19:38.321114 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.320968 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:19:38.321114 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.321070 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:19:38.321401 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.321362 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:19:38.322937 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.322903 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:19:38.322937 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.322931 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:19:38.323194 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.323165 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-kube-api-access-vfdjh" (OuterVolumeSpecName: "kube-api-access-vfdjh") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "kube-api-access-vfdjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:19:38.323194 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.323181 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" (UID: "88ceec98-6c45-4e4d-a3e9-c6f1c555adaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:19:38.422263 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422233 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfdjh\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-kube-api-access-vfdjh\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.422263 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422259 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.422263 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422269 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.422494 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422277 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.422494 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422287 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.422494 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422294 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.422494 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422309 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:38.422494 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:38.422318 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:19:39.134531 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:39.134433 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" event={"ID":"88ceec98-6c45-4e4d-a3e9-c6f1c555adaf","Type":"ContainerDied","Data":"f9f2d09100ec08265973aff10d60ddb42595593497fb99bbb08c10afeb7b6bc3"} Jul 13 15:19:39.134531 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:39.134493 2571 scope.go:117] "RemoveContainer" containerID="2ae74628911f36e027d7814c7a99dbdbc77abcd868f978fa20937fb79c224c83" Jul 13 15:19:39.134531 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:39.134496 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms" Jul 13 15:19:39.157382 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:39.157352 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms"] Jul 13 15:19:39.163996 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:39.163970 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d5c92a-openshift-default-867bc668d5-kwpms"] Jul 13 15:19:40.870940 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:40.870905 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" path="/var/lib/kubelet/pods/88ceec98-6c45-4e4d-a3e9-c6f1c555adaf/volumes" Jul 13 15:19:41.717819 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.717782 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b"] Jul 13 15:19:41.718438 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.718420 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" Jul 13 15:19:41.718508 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.718441 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" Jul 13 15:19:41.718580 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.718569 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" Jul 13 15:19:41.718617 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.718583 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" Jul 13 15:19:41.723497 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.723473 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.726332 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.726306 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-425f46-openshift-default-dockercfg-xl88l\"" Jul 13 15:19:41.738684 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.738650 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b"] Jul 13 15:19:41.855365 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855330 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-credential-socket\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855536 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855389 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-envoy\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855536 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-socket\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855536 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855513 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-podinfo\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855646 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-data\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855646 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855605 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662f37a-6edf-4e9b-baff-b4a822705e67-istiod-ca-cert\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855646 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855627 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-token\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-certs\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.855783 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.855757 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbw2\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-kube-api-access-ghbw2\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956340 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-podinfo\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956395 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-data\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662f37a-6edf-4e9b-baff-b4a822705e67-istiod-ca-cert\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-token\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-certs\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956558 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbw2\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-kube-api-access-ghbw2\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-credential-socket\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-envoy\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956699 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-socket\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.956813 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956792 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-data\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.957302 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.956950 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-certs\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.957302 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.957127 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-socket\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.957302 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.957262 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-credential-socket\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.957302 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.957268 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662f37a-6edf-4e9b-baff-b4a822705e67-istiod-ca-cert\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.958970 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.958948 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-podinfo\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.959162 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.959132 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-envoy\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.965296 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.965274 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbw2\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-kube-api-access-ghbw2\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:41.966387 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:41.966367 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-token\") pod \"e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:42.035534 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:42.035497 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:42.192691 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:42.192663 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b"] Jul 13 15:19:42.194502 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:19:42.194475 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb662f37a_6edf_4e9b_baff_b4a822705e67.slice/crio-f47e572cabc0054ce5b70ee10f9d3409471ea5bb194f2d11195733c5dc5aedf0 WatchSource:0}: Error finding container f47e572cabc0054ce5b70ee10f9d3409471ea5bb194f2d11195733c5dc5aedf0: Status 404 returned error can't find the container with id f47e572cabc0054ce5b70ee10f9d3409471ea5bb194f2d11195733c5dc5aedf0 Jul 13 15:19:42.196596 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:42.196561 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:42.196679 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:42.196633 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:42.196679 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:42.196663 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:43.157397 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:43.157357 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" event={"ID":"b662f37a-6edf-4e9b-baff-b4a822705e67","Type":"ContainerStarted","Data":"1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a"} Jul 13 15:19:43.157885 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:43.157417 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" event={"ID":"b662f37a-6edf-4e9b-baff-b4a822705e67","Type":"ContainerStarted","Data":"f47e572cabc0054ce5b70ee10f9d3409471ea5bb194f2d11195733c5dc5aedf0"} Jul 13 15:19:43.181176 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:43.181050 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" podStartSLOduration=2.18102772 podStartE2EDuration="2.18102772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:19:43.180220467 +0000 UTC m=+2020.834186924" watchObservedRunningTime="2026-07-13 15:19:43.18102772 +0000 UTC m=+2020.834994175" Jul 13 15:19:44.035825 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:44.035781 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:44.042743 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:44.042704 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:44.163230 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:44.163202 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:44.164086 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:44.164068 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:19:50.408665 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.408628 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:19:50.623272 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.623237 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc"] Jul 13 15:19:50.623867 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.623851 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" Jul 13 15:19:50.623936 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.623870 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ceec98-6c45-4e4d-a3e9-c6f1c555adaf" containerName="istio-proxy" Jul 13 15:19:50.628531 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.628512 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.632206 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.632180 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0ac978-openshift-default-dockercfg-pwvl9\"" Jul 13 15:19:50.640664 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.640633 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc"] Jul 13 15:19:50.740532 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740443 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-certs\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740532 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zpr\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-kube-api-access-w6zpr\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740549 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-credential-socket\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740574 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-token\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istiod-ca-cert\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740629 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-socket\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740748 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740708 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-envoy\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740942 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740755 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-data\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.740942 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.740771 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-podinfo\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.841653 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841611 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-certs\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.841856 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zpr\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-kube-api-access-w6zpr\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.841856 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-credential-socket\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.841856 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-token\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.841856 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841774 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istiod-ca-cert\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.841856 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841806 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-socket\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842190 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-envoy\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842190 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-data\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842190 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.841938 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-podinfo\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842190 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.842094 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-certs\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842685 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.842656 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-credential-socket\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842816 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.842741 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istiod-ca-cert\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842816 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.842756 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-socket\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.842816 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.842808 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-data\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.845120 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.845097 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-envoy\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.845457 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.845435 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-podinfo\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.851652 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.851620 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-token\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.851760 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.851723 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zpr\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-kube-api-access-w6zpr\") pod \"e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:50.943127 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:50.943083 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:51.095844 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.095812 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc"] Jul 13 15:19:51.098531 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:19:51.098495 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecc4d21_bc16_4f0b_baa3_726d9bc89866.slice/crio-e5547f774583794e1dfff3da3cc34a7df92fc305dbb15f7a1fbc21c9800e1754 WatchSource:0}: Error finding container e5547f774583794e1dfff3da3cc34a7df92fc305dbb15f7a1fbc21c9800e1754: Status 404 returned error can't find the container with id e5547f774583794e1dfff3da3cc34a7df92fc305dbb15f7a1fbc21c9800e1754 Jul 13 15:19:51.100707 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.100674 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:51.100803 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.100737 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:51.100803 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.100763 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:51.206555 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.206511 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" event={"ID":"1ecc4d21-bc16-4f0b-baa3-726d9bc89866","Type":"ContainerStarted","Data":"426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5"} Jul 13 15:19:51.206745 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.206587 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" event={"ID":"1ecc4d21-bc16-4f0b-baa3-726d9bc89866","Type":"ContainerStarted","Data":"e5547f774583794e1dfff3da3cc34a7df92fc305dbb15f7a1fbc21c9800e1754"} Jul 13 15:19:51.228287 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.228227 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" podStartSLOduration=1.228205135 podStartE2EDuration="1.228205135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:19:51.227079732 +0000 UTC m=+2028.881046189" watchObservedRunningTime="2026-07-13 15:19:51.228205135 +0000 UTC m=+2028.882171590" Jul 13 15:19:51.943777 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:51.943743 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:52.957702 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:52.957670 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:53.217738 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:53.217651 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:53.218542 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:53.218525 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:19:59.164891 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.164852 2571 patch_prober.go:28] interesting pod/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 13 15:19:59.165446 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.164919 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 13 15:19:59.254956 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.254921 2571 generic.go:356] "Generic (PLEG): container finished" podID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerID="1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a" exitCode=0 Jul 13 15:19:59.255132 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.254993 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" event={"ID":"b662f37a-6edf-4e9b-baff-b4a822705e67","Type":"ContainerDied","Data":"1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a"} Jul 13 15:19:59.255469 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.255454 2571 scope.go:117] "RemoveContainer" containerID="1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a" Jul 13 15:19:59.256581 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.256553 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:59.256656 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.256620 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:19:59.256705 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:19:59.256652 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:00.262252 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:00.262219 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" event={"ID":"b662f37a-6edf-4e9b-baff-b4a822705e67","Type":"ContainerStarted","Data":"0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d"} Jul 13 15:20:01.036103 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:01.036066 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:20:01.036679 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:01.036651 2571 patch_prober.go:28] interesting pod/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 13 15:20:01.036871 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:01.036846 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 13 15:20:01.101434 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:01.101397 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:20:02.036147 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:02.036107 2571 patch_prober.go:28] interesting pod/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 13 15:20:02.036792 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:02.036753 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 13 15:20:03.036790 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:03.036750 2571 patch_prober.go:28] interesting pod/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 13 15:20:03.037418 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:03.037383 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 13 15:20:04.037082 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:04.036610 2571 patch_prober.go:28] interesting pod/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 13 15:20:04.037082 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:04.036692 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 13 15:20:05.048793 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:05.048747 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:20:05.049641 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:05.049614 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:20:05.053793 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:05.053770 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:20:09.259310 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:09.259267 2571 patch_prober.go:28] interesting pod/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 15:20:09.259702 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:09.259341 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 15:20:10.324100 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:10.324068 2571 generic.go:356] "Generic (PLEG): container finished" podID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerID="426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5" exitCode=0 Jul 13 15:20:10.324554 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:10.324141 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" event={"ID":"1ecc4d21-bc16-4f0b-baa3-726d9bc89866","Type":"ContainerDied","Data":"426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5"} Jul 13 15:20:10.324601 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:10.324568 2571 scope.go:117] "RemoveContainer" containerID="426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5" Jul 13 15:20:10.325641 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:10.325609 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:10.326187 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:10.326118 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:10.326311 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:10.326283 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:11.330627 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:11.330589 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" event={"ID":"1ecc4d21-bc16-4f0b-baa3-726d9bc89866","Type":"ContainerStarted","Data":"500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be"} Jul 13 15:20:11.944395 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:11.943956 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:20:11.944395 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:11.944081 2571 patch_prober.go:28] interesting pod/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 13 15:20:11.944395 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:11.944127 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 13 15:20:12.943631 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:12.943590 2571 patch_prober.go:28] interesting pod/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 13 15:20:12.944268 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:12.944234 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 13 15:20:13.944173 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:13.944116 2571 patch_prober.go:28] interesting pod/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 13 15:20:13.944756 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:13.944724 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 13 15:20:13.998391 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:13.998356 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b"] Jul 13 15:20:13.998562 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:13.998545 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" containerID="cri-o://0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d" gracePeriod=30 Jul 13 15:20:14.944064 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:14.943643 2571 patch_prober.go:28] interesting pod/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 13 15:20:14.944064 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:14.943704 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 13 15:20:15.947181 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:15.947124 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:20:15.947681 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:15.947475 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:20:15.948116 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:15.948091 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:20:17.417018 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:17.416988 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:20:19.270022 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.269954 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:20:19.350411 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350372 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-socket\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350598 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350437 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghbw2\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-kube-api-access-ghbw2\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350598 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350467 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-credential-socket\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350598 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350507 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-token\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350598 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350563 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-envoy\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350598 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350592 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-certs\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350874 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350649 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662f37a-6edf-4e9b-baff-b4a822705e67-istiod-ca-cert\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350874 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350685 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-podinfo\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350874 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350695 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:19.350874 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350716 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-data\") pod \"b662f37a-6edf-4e9b-baff-b4a822705e67\" (UID: \"b662f37a-6edf-4e9b-baff-b4a822705e67\") " Jul 13 15:20:19.350874 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.350850 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:19.351143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.351111 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:19.351143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.351123 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.351291 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.351195 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.351382 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.351357 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b662f37a-6edf-4e9b-baff-b4a822705e67-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:20:19.351689 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.351665 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-data" (OuterVolumeSpecName: "istio-data") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:19.352878 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.352857 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-token" (OuterVolumeSpecName: "istio-token") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:20:19.352967 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.352933 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-kube-api-access-ghbw2" (OuterVolumeSpecName: "kube-api-access-ghbw2") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "kube-api-access-ghbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:20:19.353179 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.353141 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:20:19.353311 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.353236 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b662f37a-6edf-4e9b-baff-b4a822705e67" (UID: "b662f37a-6edf-4e9b-baff-b4a822705e67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:19.381793 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.381756 2571 generic.go:356] "Generic (PLEG): container finished" podID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerID="0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d" exitCode=0 Jul 13 15:20:19.381972 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.381805 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" Jul 13 15:20:19.381972 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.381845 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" event={"ID":"b662f37a-6edf-4e9b-baff-b4a822705e67","Type":"ContainerDied","Data":"0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d"} Jul 13 15:20:19.381972 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.381904 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b" event={"ID":"b662f37a-6edf-4e9b-baff-b4a822705e67","Type":"ContainerDied","Data":"f47e572cabc0054ce5b70ee10f9d3409471ea5bb194f2d11195733c5dc5aedf0"} Jul 13 15:20:19.381972 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.381926 2571 scope.go:117] "RemoveContainer" containerID="0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d" Jul 13 15:20:19.391949 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.391926 2571 scope.go:117] "RemoveContainer" containerID="1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a" Jul 13 15:20:19.400761 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.400737 2571 scope.go:117] "RemoveContainer" containerID="0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d" Jul 13 15:20:19.401063 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:20:19.401038 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d\": container with ID starting with 0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d not found: ID does not exist" containerID="0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d" Jul 13 15:20:19.401141 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.401072 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d"} err="failed to get container status \"0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d\": rpc error: code = NotFound desc = could not find container \"0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d\": container with ID starting with 0db0d725213b880c202f694776437575ebe2954702ed96607ffa8c90eee40c6d not found: ID does not exist" Jul 13 15:20:19.401141 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.401089 2571 scope.go:117] "RemoveContainer" containerID="1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a" Jul 13 15:20:19.401359 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:20:19.401344 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a\": container with ID starting with 1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a not found: ID does not exist" containerID="1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a" Jul 13 15:20:19.401402 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.401362 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a"} err="failed to get container status \"1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a\": rpc error: code = NotFound desc = could not find container \"1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a\": container with ID starting with 1703d5f7227c9ed452cefc2bae96f447209934fa72a9b14dea20e3692a01b96a not found: ID does not exist" Jul 13 15:20:19.412681 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.412646 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b"] Jul 13 15:20:19.418639 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.418612 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-425f46-openshift-default-745c7bb45c-95b2b"] Jul 13 15:20:19.452529 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.452494 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.452529 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.452522 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.452529 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.452530 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbw2\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-kube-api-access-ghbw2\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.452770 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.452541 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.452770 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.452551 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.452770 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.452558 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b662f37a-6edf-4e9b-baff-b4a822705e67-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:19.452770 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:19.452567 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b662f37a-6edf-4e9b-baff-b4a822705e67-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:20.871543 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:20.871506 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" path="/var/lib/kubelet/pods/b662f37a-6edf-4e9b-baff-b4a822705e67/volumes" Jul 13 15:20:21.424257 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:21.424219 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc"] Jul 13 15:20:21.424539 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:21.424511 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" containerID="cri-o://500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be" gracePeriod=30 Jul 13 15:20:22.400494 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.400455 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp"] Jul 13 15:20:22.401115 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.401089 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" Jul 13 15:20:22.401342 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.401319 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" Jul 13 15:20:22.401529 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.401513 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" Jul 13 15:20:22.401643 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.401559 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" Jul 13 15:20:22.406283 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.406256 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.409406 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.409385 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-0a6921-openshift-default-dockercfg-nbg5h\"" Jul 13 15:20:22.415962 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.415927 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp"] Jul 13 15:20:22.485601 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485554 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-certs\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.485789 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485621 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5xd\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-kube-api-access-5g5xd\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.485789 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47022f89-cff6-4c50-b6d7-213452f5646d-istiod-ca-cert\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.485789 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485673 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47022f89-cff6-4c50-b6d7-213452f5646d-istio-podinfo\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.485789 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-socket\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.485789 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485750 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-data\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.485789 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-istio-token\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.486143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-envoy\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.486143 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.485975 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-credential-socket\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587265 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587226 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-data\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587458 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-istio-token\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587458 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587348 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-envoy\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587458 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587376 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-credential-socket\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587458 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-certs\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587465 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5xd\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-kube-api-access-5g5xd\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47022f89-cff6-4c50-b6d7-213452f5646d-istiod-ca-cert\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47022f89-cff6-4c50-b6d7-213452f5646d-istio-podinfo\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587696 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587552 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-socket\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.587927 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.587697 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-data\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.588359 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.588307 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-certs\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.588359 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.588332 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47022f89-cff6-4c50-b6d7-213452f5646d-istiod-ca-cert\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.588542 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.588336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-credential-socket\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.588598 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.588536 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-socket\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.590462 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.590426 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-envoy\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.590692 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.590666 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47022f89-cff6-4c50-b6d7-213452f5646d-istio-podinfo\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.600916 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.600888 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-istio-token\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.601050 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.601019 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5xd\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-kube-api-access-5g5xd\") pod \"e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.719338 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.719237 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:22.871781 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:20:22.871749 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47022f89_cff6_4c50_b6d7_213452f5646d.slice/crio-1104cf8926da1d558bd3dbe646a05a9aac2053d3ea197fe464e45b53ce63f5c3 WatchSource:0}: Error finding container 1104cf8926da1d558bd3dbe646a05a9aac2053d3ea197fe464e45b53ce63f5c3: Status 404 returned error can't find the container with id 1104cf8926da1d558bd3dbe646a05a9aac2053d3ea197fe464e45b53ce63f5c3 Jul 13 15:20:22.873137 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.873117 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp"] Jul 13 15:20:22.873825 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.873793 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:22.873894 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.873876 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:22.873935 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:22.873904 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:23.406349 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:23.406295 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" event={"ID":"47022f89-cff6-4c50-b6d7-213452f5646d","Type":"ContainerStarted","Data":"a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb"} Jul 13 15:20:23.406759 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:23.406369 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" event={"ID":"47022f89-cff6-4c50-b6d7-213452f5646d","Type":"ContainerStarted","Data":"1104cf8926da1d558bd3dbe646a05a9aac2053d3ea197fe464e45b53ce63f5c3"} Jul 13 15:20:23.433530 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:23.433484 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podStartSLOduration=1.433468434 podStartE2EDuration="1.433468434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:20:23.431729094 +0000 UTC m=+2061.085695547" watchObservedRunningTime="2026-07-13 15:20:23.433468434 +0000 UTC m=+2061.087434947" Jul 13 15:20:23.720444 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:23.720344 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:23.725451 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:23.725426 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:24.411504 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:24.411471 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:24.412582 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:24.412558 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:26.701588 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.701560 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:20:26.830218 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830179 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-token\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830218 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830221 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-credential-socket\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830467 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830263 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-envoy\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830467 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830311 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6zpr\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-kube-api-access-w6zpr\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830467 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830335 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-data\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830467 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830358 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istiod-ca-cert\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830467 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830418 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-socket\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830467 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830446 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-podinfo\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830773 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830503 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-certs\") pod \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\" (UID: \"1ecc4d21-bc16-4f0b-baa3-726d9bc89866\") " Jul 13 15:20:26.830773 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830603 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:26.830880 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830821 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:26.830880 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.830847 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.831034 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.831007 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:20:26.831101 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.831037 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:26.831496 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.831475 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:26.832604 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.832586 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:20:26.832688 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.832641 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:20:26.832958 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.832935 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:26.832958 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.832948 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-kube-api-access-w6zpr" (OuterVolumeSpecName: "kube-api-access-w6zpr") pod "1ecc4d21-bc16-4f0b-baa3-726d9bc89866" (UID: "1ecc4d21-bc16-4f0b-baa3-726d9bc89866"). InnerVolumeSpecName "kube-api-access-w6zpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:20:26.932144 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932110 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zpr\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-kube-api-access-w6zpr\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.932144 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932142 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.932347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932166 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.932347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932178 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.932347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932187 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.932347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932197 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.932347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932204 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:26.932347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:26.932211 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecc4d21-bc16-4f0b-baa3-726d9bc89866-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:20:27.428793 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.428757 2571 generic.go:356] "Generic (PLEG): container finished" podID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerID="500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be" exitCode=0 Jul 13 15:20:27.429009 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.428820 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" Jul 13 15:20:27.429009 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.428846 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" event={"ID":"1ecc4d21-bc16-4f0b-baa3-726d9bc89866","Type":"ContainerDied","Data":"500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be"} Jul 13 15:20:27.429009 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.428902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc" event={"ID":"1ecc4d21-bc16-4f0b-baa3-726d9bc89866","Type":"ContainerDied","Data":"e5547f774583794e1dfff3da3cc34a7df92fc305dbb15f7a1fbc21c9800e1754"} Jul 13 15:20:27.429009 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.428924 2571 scope.go:117] "RemoveContainer" containerID="500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be" Jul 13 15:20:27.438492 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.438472 2571 scope.go:117] "RemoveContainer" containerID="426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5" Jul 13 15:20:27.449579 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.449554 2571 scope.go:117] "RemoveContainer" containerID="500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be" Jul 13 15:20:27.450181 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:20:27.450133 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be\": container with ID starting with 500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be not found: ID does not exist" containerID="500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be" Jul 13 15:20:27.450331 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.450188 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be"} err="failed to get container status \"500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be\": rpc error: code = NotFound desc = could not find container \"500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be\": container with ID starting with 500e845bf8df9640d4f508233af11248ab6ac3e2dc9fbbf40e1493af705d22be not found: ID does not exist" Jul 13 15:20:27.450331 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.450208 2571 scope.go:117] "RemoveContainer" containerID="426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5" Jul 13 15:20:27.450526 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:20:27.450505 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5\": container with ID starting with 426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5 not found: ID does not exist" containerID="426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5" Jul 13 15:20:27.450585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.450532 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5"} err="failed to get container status \"426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5\": rpc error: code = NotFound desc = could not find container \"426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5\": container with ID starting with 426dffd864cacf1900098b82937e8dde6ca48b7b7cb0767ed46538e55caedac5 not found: ID does not exist" Jul 13 15:20:27.458587 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.458564 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc"] Jul 13 15:20:27.465616 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:27.465591 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0ac978-openshift-default-548cc9975c-fd4bc"] Jul 13 15:20:28.870762 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:28.870727 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" path="/var/lib/kubelet/pods/1ecc4d21-bc16-4f0b-baa3-726d9bc89866/volumes" Jul 13 15:20:30.585765 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.585729 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr"] Jul 13 15:20:30.586291 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.586183 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" Jul 13 15:20:30.586291 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.586205 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" Jul 13 15:20:30.586291 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.586223 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" Jul 13 15:20:30.586291 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.586229 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b662f37a-6edf-4e9b-baff-b4a822705e67" containerName="istio-proxy" Jul 13 15:20:30.586465 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.586315 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" Jul 13 15:20:30.591173 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.591136 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.593907 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.593883 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-215a0a-openshift-default-dockercfg-szz7z\"" Jul 13 15:20:30.603886 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.603861 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr"] Jul 13 15:20:30.769678 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.769640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-certs\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.769841 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.769692 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-istio-token\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.769841 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.769746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-envoy\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.769967 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.769860 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-socket\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.769967 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.769881 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-data\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.769967 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.769903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg4s\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-kube-api-access-shg4s\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.769967 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.769927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58656570-a08e-436f-9219-9a94924db24c-istio-podinfo\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.770105 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.770010 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-credential-socket\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.770105 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.770035 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58656570-a08e-436f-9219-9a94924db24c-istiod-ca-cert\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870618 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870525 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-credential-socket\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870618 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58656570-a08e-436f-9219-9a94924db24c-istiod-ca-cert\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870858 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870618 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-certs\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870858 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-istio-token\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870858 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870691 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-envoy\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870858 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870759 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-socket\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870858 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870810 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-data\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.870858 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870847 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg4s\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-kube-api-access-shg4s\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.871168 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870881 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58656570-a08e-436f-9219-9a94924db24c-istio-podinfo\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.871168 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.870957 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-credential-socket\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.871168 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.871030 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-certs\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.871168 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.871134 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-socket\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.871442 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.871421 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-data\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.871613 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.871596 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58656570-a08e-436f-9219-9a94924db24c-istiod-ca-cert\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.873033 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.873016 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-envoy\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.873402 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.873378 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58656570-a08e-436f-9219-9a94924db24c-istio-podinfo\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.885835 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.885799 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-istio-token\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.887855 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.887832 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg4s\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-kube-api-access-shg4s\") pod \"e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:30.901778 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:30.901745 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:31.059504 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.059471 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr"] Jul 13 15:20:31.061065 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:20:31.061040 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58656570_a08e_436f_9219_9a94924db24c.slice/crio-7a6561f34ab50243984bf0555f231f11b2337fe03ee4e2b2186090a3d3a4f739 WatchSource:0}: Error finding container 7a6561f34ab50243984bf0555f231f11b2337fe03ee4e2b2186090a3d3a4f739: Status 404 returned error can't find the container with id 7a6561f34ab50243984bf0555f231f11b2337fe03ee4e2b2186090a3d3a4f739 Jul 13 15:20:31.070104 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.063704 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:31.070104 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.063870 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:31.070104 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.063962 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:31.400237 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.400139 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:20:31.454356 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.454066 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" event={"ID":"58656570-a08e-436f-9219-9a94924db24c","Type":"ContainerStarted","Data":"a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0"} Jul 13 15:20:31.454356 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.454173 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" event={"ID":"58656570-a08e-436f-9219-9a94924db24c","Type":"ContainerStarted","Data":"7a6561f34ab50243984bf0555f231f11b2337fe03ee4e2b2186090a3d3a4f739"} Jul 13 15:20:31.478715 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.478661 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" podStartSLOduration=1.478643213 podStartE2EDuration="1.478643213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:20:31.476580678 +0000 UTC m=+2069.130547135" watchObservedRunningTime="2026-07-13 15:20:31.478643213 +0000 UTC m=+2069.132609667" Jul 13 15:20:31.901914 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:31.901871 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:32.907529 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:32.907494 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:33.468372 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:33.468345 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:33.469374 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:33.469348 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:39.413267 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.413177 2571 patch_prober.go:28] interesting pod/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 13 15:20:39.413267 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.413244 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 13 15:20:39.503278 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.503192 2571 generic.go:356] "Generic (PLEG): container finished" podID="47022f89-cff6-4c50-b6d7-213452f5646d" containerID="a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb" exitCode=0 Jul 13 15:20:39.503428 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.503267 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" event={"ID":"47022f89-cff6-4c50-b6d7-213452f5646d","Type":"ContainerDied","Data":"a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb"} Jul 13 15:20:39.503703 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.503683 2571 scope.go:117] "RemoveContainer" containerID="a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb" Jul 13 15:20:39.504717 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.504687 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:39.504780 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.504756 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:39.504824 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:39.504791 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:40.510995 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:40.510960 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" event={"ID":"47022f89-cff6-4c50-b6d7-213452f5646d","Type":"ContainerStarted","Data":"1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685"} Jul 13 15:20:40.720068 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:40.720028 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:40.720391 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:40.720367 2571 patch_prober.go:28] interesting pod/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 13 15:20:40.720443 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:40.720419 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 13 15:20:41.720534 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:41.720498 2571 patch_prober.go:28] interesting pod/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 13 15:20:41.720956 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:41.720580 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 13 15:20:42.719688 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:42.719654 2571 patch_prober.go:28] interesting pod/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 13 15:20:42.719867 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:42.719719 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 13 15:20:43.720481 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:43.720433 2571 patch_prober.go:28] interesting pod/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 13 15:20:43.720905 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:43.720506 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 13 15:20:44.720387 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:44.720347 2571 patch_prober.go:28] interesting pod/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 13 15:20:44.720562 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:44.720438 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 13 15:20:45.724371 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:45.724337 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:45.724825 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:45.724638 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:45.725277 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:45.725253 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:20:46.549127 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:46.549093 2571 generic.go:356] "Generic (PLEG): container finished" podID="58656570-a08e-436f-9219-9a94924db24c" containerID="a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0" exitCode=0 Jul 13 15:20:46.549383 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:46.549181 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" event={"ID":"58656570-a08e-436f-9219-9a94924db24c","Type":"ContainerDied","Data":"a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0"} Jul 13 15:20:46.549582 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:46.549566 2571 scope.go:117] "RemoveContainer" containerID="a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0" Jul 13 15:20:46.550554 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:46.550525 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:46.550608 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:46.550593 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:46.550644 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:46.550626 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 15:20:46.996459 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:46.996375 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:20:47.558034 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:47.557994 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" event={"ID":"58656570-a08e-436f-9219-9a94924db24c","Type":"ContainerStarted","Data":"8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a"} Jul 13 15:20:47.902593 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:47.902499 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:47.903741 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:47.903715 2571 patch_prober.go:28] interesting pod/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 15:20:47.903811 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:47.903770 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 15:20:48.468308 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:48.468263 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:48.903128 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:48.903091 2571 patch_prober.go:28] interesting pod/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 15:20:48.903343 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:48.903178 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 15:20:49.902790 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:49.902748 2571 patch_prober.go:28] interesting pod/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 15:20:49.903230 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:49.902811 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 15:20:50.902789 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:50.902743 2571 patch_prober.go:28] interesting pod/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 15:20:50.902974 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:50.902804 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 15:20:51.902131 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:51.902095 2571 patch_prober.go:28] interesting pod/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 15:20:51.902347 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:51.902172 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 15:20:52.906484 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:52.906453 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:52.907409 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:52.907389 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:20:56.809689 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:20:56.809648 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:21:02.973034 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:21:02.973005 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:21:02.980931 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:21:02.980908 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:21:12.900747 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:21:12.900706 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:21:29.523713 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:21:29.523672 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:21:44.004068 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:21:44.004032 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:21:58.399076 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:21:58.398990 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:22:14.704894 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:22:14.704854 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:22:29.806968 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:22:29.806926 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:22:46.702107 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:22:46.702057 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:23:04.026266 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:23:04.026228 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:23:18.799987 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:23:18.799948 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:23:51.810318 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:23:51.810274 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:24:19.110050 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:24:19.110006 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:24:37.603509 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:24:37.603468 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:24:55.305814 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:24:55.305733 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:25:05.821666 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:25:05.821625 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:25:26.099660 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:25:26.099620 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:25:39.409686 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:25:39.409642 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:25:54.006078 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:25:54.006036 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:26:03.010174 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:03.010118 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:26:03.018307 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:03.018276 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:26:07.308997 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:07.308955 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:26:45.733236 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:45.733204 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp"] Jul 13 15:26:45.733783 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:45.733412 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" containerID="cri-o://1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685" gracePeriod=30 Jul 13 15:26:49.505880 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:49.505845 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ndlnp"] Jul 13 15:26:50.990001 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:50.989975 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:26:51.056095 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056070 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47022f89-cff6-4c50-b6d7-213452f5646d-istio-podinfo\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056303 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056117 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47022f89-cff6-4c50-b6d7-213452f5646d-istiod-ca-cert\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056303 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056145 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-credential-socket\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056303 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056197 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-data\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056303 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056218 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-certs\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056303 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056257 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-envoy\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056303 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056274 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-socket\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056303 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056292 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g5xd\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-kube-api-access-5g5xd\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056702 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056386 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-istio-token\") pod \"47022f89-cff6-4c50-b6d7-213452f5646d\" (UID: \"47022f89-cff6-4c50-b6d7-213452f5646d\") " Jul 13 15:26:51.056702 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056472 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:26:51.056702 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056645 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47022f89-cff6-4c50-b6d7-213452f5646d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:26:51.058055 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056840 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:26:51.058055 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056899 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47022f89-cff6-4c50-b6d7-213452f5646d-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.058055 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056919 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.058055 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056930 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:26:51.058055 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.056966 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-data" (OuterVolumeSpecName: "istio-data") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:26:51.059699 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.059672 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-istio-token" (OuterVolumeSpecName: "istio-token") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:26:51.059942 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.059907 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-kube-api-access-5g5xd" (OuterVolumeSpecName: "kube-api-access-5g5xd") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "kube-api-access-5g5xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:26:51.060283 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.060249 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:26:51.060725 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.060673 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47022f89-cff6-4c50-b6d7-213452f5646d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47022f89-cff6-4c50-b6d7-213452f5646d" (UID: "47022f89-cff6-4c50-b6d7-213452f5646d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:26:51.157487 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.157457 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47022f89-cff6-4c50-b6d7-213452f5646d-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.157487 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.157483 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.157487 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.157493 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.157726 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.157501 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.157726 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.157510 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47022f89-cff6-4c50-b6d7-213452f5646d-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.157726 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.157518 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5xd\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-kube-api-access-5g5xd\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.157726 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.157526 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47022f89-cff6-4c50-b6d7-213452f5646d-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:26:51.518511 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.518426 2571 generic.go:356] "Generic (PLEG): container finished" podID="47022f89-cff6-4c50-b6d7-213452f5646d" containerID="1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685" exitCode=0 Jul 13 15:26:51.518511 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.518474 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" Jul 13 15:26:51.518691 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.518511 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" event={"ID":"47022f89-cff6-4c50-b6d7-213452f5646d","Type":"ContainerDied","Data":"1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685"} Jul 13 15:26:51.518691 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.518566 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp" event={"ID":"47022f89-cff6-4c50-b6d7-213452f5646d","Type":"ContainerDied","Data":"1104cf8926da1d558bd3dbe646a05a9aac2053d3ea197fe464e45b53ce63f5c3"} Jul 13 15:26:51.518691 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.518589 2571 scope.go:117] "RemoveContainer" containerID="1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685" Jul 13 15:26:51.528202 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.528185 2571 scope.go:117] "RemoveContainer" containerID="a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb" Jul 13 15:26:51.536558 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.536532 2571 scope.go:117] "RemoveContainer" containerID="1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685" Jul 13 15:26:51.536874 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:26:51.536845 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685\": container with ID starting with 1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685 not found: ID does not exist" containerID="1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685" Jul 13 15:26:51.536974 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.536886 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685"} err="failed to get container status \"1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685\": rpc error: code = NotFound desc = could not find container \"1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685\": container with ID starting with 1d7df39f8361443d8bb0b42e750a4207d5b17a22732319fe057df244c4cdf685 not found: ID does not exist" Jul 13 15:26:51.536974 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.536912 2571 scope.go:117] "RemoveContainer" containerID="a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb" Jul 13 15:26:51.537270 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:26:51.537191 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb\": container with ID starting with a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb not found: ID does not exist" containerID="a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb" Jul 13 15:26:51.537270 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.537223 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb"} err="failed to get container status \"a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb\": rpc error: code = NotFound desc = could not find container \"a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb\": container with ID starting with a606f9ce493debe167b11b8104722be99d5eaf7f7afd26aa14712080f6a7d3cb not found: ID does not exist" Jul 13 15:26:51.559712 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.559685 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp"] Jul 13 15:26:51.563876 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:51.563852 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-0a6921-openshift-default-7bc84d64b7-2l6gp"] Jul 13 15:26:52.870901 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:52.870869 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" path="/var/lib/kubelet/pods/47022f89-cff6-4c50-b6d7-213452f5646d/volumes" Jul 13 15:26:54.642673 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:54.642641 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr"] Jul 13 15:26:54.643200 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:54.642911 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" containerID="cri-o://8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a" gracePeriod=30 Jul 13 15:26:55.313802 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:55.313767 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fdd6798bf-g7lst_4cc28573-d530-4efc-8c71-d935fe5a937e/authorino/0.log" Jul 13 15:26:59.913650 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:26:59.913583 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:27:00.044230 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044179 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-credential-socket\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044230 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044238 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-istio-token\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044264 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-data\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044299 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58656570-a08e-436f-9219-9a94924db24c-istio-podinfo\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044342 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-envoy\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044362 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58656570-a08e-436f-9219-9a94924db24c-istiod-ca-cert\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044388 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-socket\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044433 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-certs\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044481 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shg4s\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-kube-api-access-shg4s\") pod \"58656570-a08e-436f-9219-9a94924db24c\" (UID: \"58656570-a08e-436f-9219-9a94924db24c\") " Jul 13 15:27:00.044878 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044530 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:27:00.044878 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044788 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:27:00.044878 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044834 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-credential-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.044878 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044867 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58656570-a08e-436f-9219-9a94924db24c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:27:00.045096 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.044998 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:27:00.045207 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.045108 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-data" (OuterVolumeSpecName: "istio-data") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:27:00.046786 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.046756 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-kube-api-access-shg4s" (OuterVolumeSpecName: "kube-api-access-shg4s") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "kube-api-access-shg4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:27:00.046925 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.046820 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58656570-a08e-436f-9219-9a94924db24c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:27:00.047126 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.047105 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-istio-token" (OuterVolumeSpecName: "istio-token") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:27:00.047221 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.047113 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58656570-a08e-436f-9219-9a94924db24c" (UID: "58656570-a08e-436f-9219-9a94924db24c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:27:00.146348 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146312 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58656570-a08e-436f-9219-9a94924db24c-istiod-ca-cert\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.146348 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146343 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-socket\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.146348 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146351 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-workload-certs\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.146585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146361 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg4s\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-kube-api-access-shg4s\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.146585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146372 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58656570-a08e-436f-9219-9a94924db24c-istio-token\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.146585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146382 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-data\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.146585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146389 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58656570-a08e-436f-9219-9a94924db24c-istio-podinfo\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.146585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.146396 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58656570-a08e-436f-9219-9a94924db24c-istio-envoy\") on node \"ip-10-0-133-41.ec2.internal\" DevicePath \"\"" Jul 13 15:27:00.215998 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.215908 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tkrt4_591cb838-631d-47f7-aad4-fe2b3f0f6d81/manager/0.log" Jul 13 15:27:00.341634 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.341603 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dh7zt_291f8a43-888a-4e3b-92a9-a6a04c6812d1/manager/0.log" Jul 13 15:27:00.569556 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.569492 2571 generic.go:356] "Generic (PLEG): container finished" podID="58656570-a08e-436f-9219-9a94924db24c" containerID="8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a" exitCode=0 Jul 13 15:27:00.569556 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.569548 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" Jul 13 15:27:00.569814 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.569576 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" event={"ID":"58656570-a08e-436f-9219-9a94924db24c","Type":"ContainerDied","Data":"8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a"} Jul 13 15:27:00.569814 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.569634 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr" event={"ID":"58656570-a08e-436f-9219-9a94924db24c","Type":"ContainerDied","Data":"7a6561f34ab50243984bf0555f231f11b2337fe03ee4e2b2186090a3d3a4f739"} Jul 13 15:27:00.569814 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.569665 2571 scope.go:117] "RemoveContainer" containerID="8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a" Jul 13 15:27:00.579469 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.579448 2571 scope.go:117] "RemoveContainer" containerID="a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0" Jul 13 15:27:00.587887 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.587863 2571 scope.go:117] "RemoveContainer" containerID="8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a" Jul 13 15:27:00.588183 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:27:00.588135 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a\": container with ID starting with 8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a not found: ID does not exist" containerID="8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a" Jul 13 15:27:00.588279 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.588191 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a"} err="failed to get container status \"8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a\": rpc error: code = NotFound desc = could not find container \"8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a\": container with ID starting with 8c6fb85771355d7185c770711108b0ec7d3e24e945cb17ae05a8b1306937340a not found: ID does not exist" Jul 13 15:27:00.588279 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.588209 2571 scope.go:117] "RemoveContainer" containerID="a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0" Jul 13 15:27:00.588477 ip-10-0-133-41 kubenswrapper[2571]: E0713 15:27:00.588457 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0\": container with ID starting with a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0 not found: ID does not exist" containerID="a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0" Jul 13 15:27:00.588520 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.588482 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0"} err="failed to get container status \"a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0\": rpc error: code = NotFound desc = could not find container \"a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0\": container with ID starting with a58b13fce0e5fe8edb0684bb13cee6d3ad8bcdad61f96799b83da98b0f202bf0 not found: ID does not exist" Jul 13 15:27:00.596060 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.595617 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr"] Jul 13 15:27:00.604500 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.604473 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-215a0a-openshift-default-85b968848c-wt8qr"] Jul 13 15:27:00.607567 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.607547 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29732580-pzndm_c3c1834c-3e8d-4f59-9d3c-59a2daff9a47/cleanup/0.log" Jul 13 15:27:00.733175 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.733121 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29732595-d5wxg_51d85393-a96c-4ac1-9013-38ce1a397313/cleanup/0.log" Jul 13 15:27:00.870962 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:00.870877 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58656570-a08e-436f-9219-9a94924db24c" path="/var/lib/kubelet/pods/58656570-a08e-436f-9219-9a94924db24c/volumes" Jul 13 15:27:01.108247 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:01.108216 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fbjkf_40821c7e-55eb-4aff-aed8-1a9fed459e5f/manager/0.log" Jul 13 15:27:01.502141 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:01.502113 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6496967d77-f9z42_dd46cd42-458f-467b-bd6d-996cb935513c/copy-manifests/0.log" Jul 13 15:27:01.515899 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:01.515875 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6496967d77-f9z42_dd46cd42-458f-467b-bd6d-996cb935513c/manager/0.log" Jul 13 15:27:02.996624 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:02.996583 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fdd6798bf-g7lst_4cc28573-d530-4efc-8c71-d935fe5a937e/authorino/0.log" Jul 13 15:27:03.124933 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:03.124905 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7dgcm_60648541-e31b-4b1b-b803-95f4a92ab6ce/manager/0.log" Jul 13 15:27:03.263844 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:03.263765 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-c9lkx_d886f0d5-ada3-4918-8536-03fd6a632ed3/manager/0.log" Jul 13 15:27:03.384690 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:03.384661 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nj4cl_841cdb67-b6ff-4426-b2ed-b149c45a4ace/kuadrant-console-plugin/0.log" Jul 13 15:27:03.499958 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:03.499928 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v5dd6_25bb007c-a58b-488e-9fbe-cdb74c52b766/registry-server/0.log" Jul 13 15:27:03.625697 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:03.625666 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qhs4s_973ed3e2-0703-47d1-96c8-3c2ae7541433/manager/0.log" Jul 13 15:27:03.745387 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:03.745359 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ndlnp_65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9/limitador/0.log" Jul 13 15:27:04.607272 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:04.607237 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84c9577b7-gtd5s_39dc3323-01b4-4bf6-91a1-6d71d1daaeea/kube-auth-proxy/0.log" Jul 13 15:27:04.736585 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:04.736558 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sjhsx_9280c649-ece0-4e47-be43-cca2b81c54b1/istio-proxy/1.log" Jul 13 15:27:04.860909 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:04.860825 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w56cq_e715bff4-309b-42fb-87af-0c7d81a0a5e6/payload-pre-processing/0.log" Jul 13 15:27:04.975707 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:04.975680 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-d46n7_92093db2-28be-4389-b673-32c7fe2f7b81/payload-processing/0.log" Jul 13 15:27:05.092297 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:05.092266 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8c9b7db-nw27s_c9cb46c8-1226-48ff-9cb5-27df8b576c32/router/0.log" Jul 13 15:27:05.603987 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:05.603957 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-85m8h_775580ba-d3ce-4fa8-864e-aaaf3ffd72ec/main/0.log" Jul 13 15:27:05.732397 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:05.732370 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sl2j6_91f0d4d0-e7ef-4078-a208-a0d8f24f8a2e/main/0.log" Jul 13 15:27:14.361450 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:14.361419 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4dphs_64eaf08c-2213-409c-8d19-7b470afb06ce/global-pull-secret-syncer/0.log" Jul 13 15:27:14.556029 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:14.555996 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kxxh2_d26ec668-e81d-41b8-9f37-48d808ce132c/konnectivity-agent/0.log" Jul 13 15:27:14.622148 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:14.622070 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-41.ec2.internal_102e87a82f37db84cece8bdb72d68ea7/haproxy/0.log" Jul 13 15:27:19.486313 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:19.486272 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fdd6798bf-g7lst_4cc28573-d530-4efc-8c71-d935fe5a937e/authorino/0.log" Jul 13 15:27:19.521944 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:19.521911 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7dgcm_60648541-e31b-4b1b-b803-95f4a92ab6ce/manager/0.log" Jul 13 15:27:19.562680 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:19.562654 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-c9lkx_d886f0d5-ada3-4918-8536-03fd6a632ed3/manager/0.log" Jul 13 15:27:19.594034 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:19.594008 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nj4cl_841cdb67-b6ff-4426-b2ed-b149c45a4ace/kuadrant-console-plugin/0.log" Jul 13 15:27:19.644353 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:19.644315 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v5dd6_25bb007c-a58b-488e-9fbe-cdb74c52b766/registry-server/0.log" Jul 13 15:27:19.743794 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:19.743697 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qhs4s_973ed3e2-0703-47d1-96c8-3c2ae7541433/manager/0.log" Jul 13 15:27:19.770430 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:19.770402 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ndlnp_65d3e1e8-ce10-4f9e-9fe6-0d14d0909ef9/limitador/0.log" Jul 13 15:27:21.645261 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.645231 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb/alertmanager/0.log" Jul 13 15:27:21.686095 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.686065 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb/config-reloader/0.log" Jul 13 15:27:21.715283 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.715258 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb/kube-rbac-proxy-web/0.log" Jul 13 15:27:21.744952 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.744929 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb/kube-rbac-proxy/0.log" Jul 13 15:27:21.782004 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.781979 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb/kube-rbac-proxy-metric/0.log" Jul 13 15:27:21.815910 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.815849 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb/prom-label-proxy/0.log" Jul 13 15:27:21.857174 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.857133 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d4f3e95-0a3d-4f3a-80bf-ad1af2c29ccb/init-config-reloader/0.log" Jul 13 15:27:21.901997 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.901963 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-gdjnc_6adfc487-32f8-44ae-a5ba-a11777af7499/cluster-monitoring-operator/0.log" Jul 13 15:27:21.939458 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.939432 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-mk6zw_e5e09e1d-0940-4bd7-b785-74e40ee40e14/kube-state-metrics/0.log" Jul 13 15:27:21.974786 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:21.974763 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-mk6zw_e5e09e1d-0940-4bd7-b785-74e40ee40e14/kube-rbac-proxy-main/0.log" Jul 13 15:27:22.015432 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:22.015395 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-mk6zw_e5e09e1d-0940-4bd7-b785-74e40ee40e14/kube-rbac-proxy-self/0.log" Jul 13 15:27:22.400236 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:22.400208 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s22nb_4c9bb98e-e2c7-4eb5-8468-2c8fda939998/node-exporter/0.log" Jul 13 15:27:22.435508 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:22.435465 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s22nb_4c9bb98e-e2c7-4eb5-8468-2c8fda939998/kube-rbac-proxy/0.log" Jul 13 15:27:22.480740 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:22.480714 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s22nb_4c9bb98e-e2c7-4eb5-8468-2c8fda939998/init-textfile/0.log" Jul 13 15:27:22.528560 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:22.528528 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-2lfc5_35d475a6-d857-4d42-917b-c0e797398243/kube-rbac-proxy-main/0.log" Jul 13 15:27:22.566037 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:22.565996 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-2lfc5_35d475a6-d857-4d42-917b-c0e797398243/kube-rbac-proxy-self/0.log" Jul 13 15:27:22.600057 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:22.600034 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-2lfc5_35d475a6-d857-4d42-917b-c0e797398243/openshift-state-metrics/0.log" Jul 13 15:27:24.507473 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.507438 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h"] Jul 13 15:27:24.508037 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508016 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" Jul 13 15:27:24.508140 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508041 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" Jul 13 15:27:24.508140 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508060 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" Jul 13 15:27:24.508140 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508068 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" Jul 13 15:27:24.508140 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508090 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" Jul 13 15:27:24.508140 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508099 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" Jul 13 15:27:24.508140 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508124 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" Jul 13 15:27:24.508140 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508132 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" Jul 13 15:27:24.508522 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508285 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" Jul 13 15:27:24.508522 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508298 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" Jul 13 15:27:24.508522 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508309 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="47022f89-cff6-4c50-b6d7-213452f5646d" containerName="istio-proxy" Jul 13 15:27:24.508522 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508318 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecc4d21-bc16-4f0b-baa3-726d9bc89866" containerName="istio-proxy" Jul 13 15:27:24.508522 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.508328 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="58656570-a08e-436f-9219-9a94924db24c" containerName="istio-proxy" Jul 13 15:27:24.511685 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.511664 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.514520 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.514497 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g54mc\"/\"default-dockercfg-f7xfv\"" Jul 13 15:27:24.514666 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.514649 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g54mc\"/\"openshift-service-ca.crt\"" Jul 13 15:27:24.515751 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.515736 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g54mc\"/\"kube-root-ca.crt\"" Jul 13 15:27:24.520032 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.520009 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h"] Jul 13 15:27:24.578812 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.578779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-sys\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.578984 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.578834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxwbc\" (UniqueName: \"kubernetes.io/projected/e3cd3253-e9df-42f4-89df-bfa29be203fc-kube-api-access-lxwbc\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.578984 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.578889 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-lib-modules\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.578984 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.578912 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-proc\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.578984 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.578941 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-podres\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680130 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-proc\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680130 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-podres\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680360 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680213 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-sys\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680360 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680221 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-proc\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680360 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxwbc\" (UniqueName: \"kubernetes.io/projected/e3cd3253-e9df-42f4-89df-bfa29be203fc-kube-api-access-lxwbc\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680360 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680276 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-sys\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680360 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680284 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-lib-modules\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680360 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680334 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-podres\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.680558 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.680390 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3cd3253-e9df-42f4-89df-bfa29be203fc-lib-modules\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.689719 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.689686 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxwbc\" (UniqueName: \"kubernetes.io/projected/e3cd3253-e9df-42f4-89df-bfa29be203fc-kube-api-access-lxwbc\") pod \"perf-node-gather-daemonset-brx2h\" (UID: \"e3cd3253-e9df-42f4-89df-bfa29be203fc\") " pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.822600 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.822570 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:24.974928 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:24.974899 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h"] Jul 13 15:27:24.977516 ip-10-0-133-41 kubenswrapper[2571]: W0713 15:27:24.977484 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode3cd3253_e9df_42f4_89df_bfa29be203fc.slice/crio-b5221630b04dfe45a7687a2560ed75b371fd4ab8577a8e3e8f67fe99c360cb81 WatchSource:0}: Error finding container b5221630b04dfe45a7687a2560ed75b371fd4ab8577a8e3e8f67fe99c360cb81: Status 404 returned error can't find the container with id b5221630b04dfe45a7687a2560ed75b371fd4ab8577a8e3e8f67fe99c360cb81 Jul 13 15:27:25.348892 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:25.348861 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/1.log" Jul 13 15:27:25.353520 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:25.353495 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-sl9b2_6c183ae1-2b41-4b01-ad48-f038e47583ed/console-operator/2.log" Jul 13 15:27:25.700431 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:25.700339 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" event={"ID":"e3cd3253-e9df-42f4-89df-bfa29be203fc","Type":"ContainerStarted","Data":"cc2e8d5fcee1e081799797d716ba5deb89ff92a24a66b49363465dd6bcba3a1c"} Jul 13 15:27:25.700431 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:25.700390 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" event={"ID":"e3cd3253-e9df-42f4-89df-bfa29be203fc","Type":"ContainerStarted","Data":"b5221630b04dfe45a7687a2560ed75b371fd4ab8577a8e3e8f67fe99c360cb81"} Jul 13 15:27:25.700842 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:25.700432 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:25.722535 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:25.722466 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" podStartSLOduration=1.722452175 podStartE2EDuration="1.722452175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 15:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:27:25.722055418 +0000 UTC m=+2483.376021874" watchObservedRunningTime="2026-07-13 15:27:25.722452175 +0000 UTC m=+2483.376418638" Jul 13 15:27:25.915739 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:25.915702 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67d65f5fdf-qbqcw_7ddd9b3d-0818-49d3-8d8f-c8d83fe1af06/console/0.log" Jul 13 15:27:26.703902 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:26.703868 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-h8xp5_f4cdb5b8-7b4d-4dd1-8588-5c0cb8b2bcc5/volume-data-source-validator/0.log" Jul 13 15:27:28.067391 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:28.067358 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kk5pz_b9027a3d-54b9-4192-a5ba-5762be52aef8/dns/0.log" Jul 13 15:27:28.092568 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:28.092539 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kk5pz_b9027a3d-54b9-4192-a5ba-5762be52aef8/kube-rbac-proxy/0.log" Jul 13 15:27:28.121469 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:28.121443 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnfks_a86f3ae9-b976-415c-abbf-956a61859697/dns-node-resolver/0.log" Jul 13 15:27:28.789589 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:28.789557 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79d4bdbdbc-cs5pk_bbc633b5-112a-4736-989d-863fa5480d0f/registry/0.log" Jul 13 15:27:28.895960 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:28.895930 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnldn_c135566f-c963-4b21-9507-4945739c5b5a/node-ca/0.log" Jul 13 15:27:30.297718 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:30.297690 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84c9577b7-gtd5s_39dc3323-01b4-4bf6-91a1-6d71d1daaeea/kube-auth-proxy/0.log" Jul 13 15:27:30.335799 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:30.335763 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sjhsx_9280c649-ece0-4e47-be43-cca2b81c54b1/istio-proxy/0.log" Jul 13 15:27:30.360515 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:30.360489 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sjhsx_9280c649-ece0-4e47-be43-cca2b81c54b1/istio-proxy/1.log" Jul 13 15:27:30.393916 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:30.393889 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w56cq_e715bff4-309b-42fb-87af-0c7d81a0a5e6/payload-pre-processing/0.log" Jul 13 15:27:30.445681 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:30.445656 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-d46n7_92093db2-28be-4389-b673-32c7fe2f7b81/payload-processing/0.log" Jul 13 15:27:30.471108 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:30.471083 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8c9b7db-nw27s_c9cb46c8-1226-48ff-9cb5-27df8b576c32/router/0.log" Jul 13 15:27:31.261090 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:31.261061 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5hbv_1c55e6c2-6bc1-4caf-b8c4-e0e654d8058e/serve-healthcheck-canary/0.log" Jul 13 15:27:31.717875 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:31.717847 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g54mc/perf-node-gather-daemonset-brx2h" Jul 13 15:27:31.942714 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:31.942681 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-crrw5_f1b26389-1c68-47e6-9201-ab14423132a7/kube-rbac-proxy/0.log" Jul 13 15:27:31.968840 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:31.968769 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-crrw5_f1b26389-1c68-47e6-9201-ab14423132a7/exporter/0.log" Jul 13 15:27:31.997238 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:31.997210 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-crrw5_f1b26389-1c68-47e6-9201-ab14423132a7/extractor/0.log" Jul 13 15:27:34.695264 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:34.695235 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tkrt4_591cb838-631d-47f7-aad4-fe2b3f0f6d81/manager/0.log" Jul 13 15:27:34.838419 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:34.838390 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dh7zt_291f8a43-888a-4e3b-92a9-a6a04c6812d1/manager/0.log" Jul 13 15:27:34.911340 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:34.911260 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29732580-pzndm_c3c1834c-3e8d-4f59-9d3c-59a2daff9a47/cleanup/0.log" Jul 13 15:27:34.939032 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:34.938996 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29732595-d5wxg_51d85393-a96c-4ac1-9013-38ce1a397313/cleanup/0.log" Jul 13 15:27:35.218794 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.218715 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fbjkf_40821c7e-55eb-4aff-aed8-1a9fed459e5f/manager/0.log" Jul 13 15:27:35.267070 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.267042 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84c9577b7-gtd5s_39dc3323-01b4-4bf6-91a1-6d71d1daaeea/kube-auth-proxy/0.log" Jul 13 15:27:35.295930 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.295899 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sjhsx_9280c649-ece0-4e47-be43-cca2b81c54b1/istio-proxy/0.log" Jul 13 15:27:35.324636 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.324611 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sjhsx_9280c649-ece0-4e47-be43-cca2b81c54b1/istio-proxy/1.log" Jul 13 15:27:35.345058 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.345035 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w56cq_e715bff4-309b-42fb-87af-0c7d81a0a5e6/payload-pre-processing/0.log" Jul 13 15:27:35.396675 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.396647 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-d46n7_92093db2-28be-4389-b673-32c7fe2f7b81/payload-processing/0.log" Jul 13 15:27:35.412257 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.412232 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6496967d77-f9z42_dd46cd42-458f-467b-bd6d-996cb935513c/manager/0.log" Jul 13 15:27:35.416788 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.416767 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8c9b7db-nw27s_c9cb46c8-1226-48ff-9cb5-27df8b576c32/router/0.log" Jul 13 15:27:35.433020 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:35.432998 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6496967d77-f9z42_dd46cd42-458f-467b-bd6d-996cb935513c/copy-manifests/0.log" Jul 13 15:27:36.845702 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:36.845672 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84c9577b7-gtd5s_39dc3323-01b4-4bf6-91a1-6d71d1daaeea/kube-auth-proxy/0.log" Jul 13 15:27:36.867976 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:36.867948 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sjhsx_9280c649-ece0-4e47-be43-cca2b81c54b1/istio-proxy/0.log" Jul 13 15:27:36.896024 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:36.896002 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sjhsx_9280c649-ece0-4e47-be43-cca2b81c54b1/istio-proxy/1.log" Jul 13 15:27:36.916578 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:36.916554 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-w56cq_e715bff4-309b-42fb-87af-0c7d81a0a5e6/payload-pre-processing/0.log" Jul 13 15:27:36.959354 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:36.959330 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-d46n7_92093db2-28be-4389-b673-32c7fe2f7b81/payload-processing/0.log" Jul 13 15:27:36.979389 ip-10-0-133-41 kubenswrapper[2571]: I0713 15:27:36.979360 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8c9b7db-nw27s_c9cb46c8-1226-48ff-9cb5-27df8b576c32/router/0.log"