Jul 29 03:55:17.586140 ip-10-0-140-47 systemd[1]: Starting Kubernetes Kubelet... Jul 29 03:55:17.971932 ip-10-0-140-47 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 03:55:17.971932 ip-10-0-140-47 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 03:55:17.971932 ip-10-0-140-47 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 03:55:17.971932 ip-10-0-140-47 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 03:55:17.971932 ip-10-0-140-47 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 03:55:17.972799 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.972718 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 03:55:17.978520 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978499 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 03:55:17.978520 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978517 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 03:55:17.978520 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978521 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 03:55:17.978520 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978524 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978528 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978531 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978535 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978537 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978540 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978543 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978546 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978548 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978552 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978554 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978557 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978560 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978562 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978565 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978567 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978570 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978572 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978575 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978578 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 03:55:17.978671 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978580 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978583 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978586 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978588 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978591 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978594 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978596 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978599 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978601 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978604 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978606 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978609 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978611 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978614 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978617 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978620 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978622 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978625 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978627 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 03:55:17.979208 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978630 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978632 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978635 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978637 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978641 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978644 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978646 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978649 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978652 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978655 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978657 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978659 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978662 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978664 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978667 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978669 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978672 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978675 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978677 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978680 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 03:55:17.979672 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978683 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978686 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978690 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978695 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978698 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978702 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978706 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978709 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978712 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978715 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978718 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978720 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978723 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978725 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978728 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978731 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978733 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978736 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978738 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 03:55:17.980160 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978741 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978745 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978749 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978752 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978756 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978759 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978762 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978764 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978767 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978769 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978772 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978775 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978777 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978780 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978783 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978785 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978787 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.978790 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979489 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979497 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 03:55:17.980612 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979501 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979504 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979507 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979509 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979512 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979514 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979517 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979519 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979522 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979524 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979527 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979530 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979532 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979535 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979537 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979540 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979543 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979546 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979548 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979550 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 03:55:17.981101 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979553 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979556 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979559 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979562 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979564 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979567 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979569 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979572 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979574 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979578 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979581 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979584 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979587 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979589 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979592 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979594 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979597 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979599 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979602 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 03:55:17.981627 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979604 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979607 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979609 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979611 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979614 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979616 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979619 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979621 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979624 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979626 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979629 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979631 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979634 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979636 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979639 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979642 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979644 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979647 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979649 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 03:55:17.982096 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979653 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979656 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979660 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979662 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979665 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979668 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979671 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979674 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979677 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979680 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979683 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979685 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979688 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979690 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979693 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979695 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979698 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979700 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979702 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979705 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 03:55:17.982549 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979707 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979710 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979712 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979715 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979718 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979720 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979724 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979727 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979730 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979732 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979735 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979737 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979740 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979742 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979745 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979747 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979750 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979752 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.979756 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 03:55:17.983037 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980200 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980206 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980209 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980212 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980215 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980218 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980220 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980223 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980226 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980228 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980231 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980233 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980236 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980239 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980241 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980244 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980247 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980250 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980252 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980255 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 03:55:17.983494 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980258 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980260 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980263 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980265 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980269 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980272 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980275 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980278 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980281 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980283 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980287 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980290 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980292 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980295 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980298 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980300 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980303 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980305 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980308 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 03:55:17.984114 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980310 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980313 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980315 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980319 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980322 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980325 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980327 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980330 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980332 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980335 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980338 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980341 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980343 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980346 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980348 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980350 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980353 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980355 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980358 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980360 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 03:55:17.984572 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980363 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980365 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980368 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980370 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980373 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980376 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980378 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980381 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980383 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980386 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980388 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980390 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980393 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980396 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980398 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980400 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980403 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980407 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980409 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980412 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 03:55:17.985074 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980414 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980417 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980419 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980422 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980425 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980427 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980431 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980434 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980437 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980440 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980443 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980445 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980448 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980450 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980453 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980455 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980457 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980460 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980462 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 03:55:17.985553 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.980465 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980530 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980537 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980544 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980549 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980555 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980561 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980568 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980574 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980582 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980585 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980588 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980592 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980595 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980597 2576 flags.go:64] FLAG: --cgroup-root="" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980600 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980603 2576 flags.go:64] FLAG: --client-ca-file="" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980607 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980610 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980614 2576 flags.go:64] FLAG: --cluster-domain="" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980617 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980620 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980623 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980626 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 03:55:17.986016 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980630 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980633 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980635 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980639 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980641 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980644 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980647 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980650 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980653 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980656 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980659 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980662 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980665 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980667 2576 flags.go:64] FLAG: --enable-server="true" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980670 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980674 2576 flags.go:64] FLAG: --event-burst="100" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980677 2576 flags.go:64] FLAG: --event-qps="50" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980681 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980684 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980686 2576 flags.go:64] FLAG: --eviction-hard="" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980700 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980703 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980706 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980709 2576 flags.go:64] FLAG: --eviction-soft="" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980712 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 03:55:17.986692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980714 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980718 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980720 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980723 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980726 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980728 2576 flags.go:64] FLAG: --feature-gates="" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980732 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980736 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980739 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980742 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980745 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980748 2576 flags.go:64] FLAG: --help="false" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980751 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-140-47.ec2.internal" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980754 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980757 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980759 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980762 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980766 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980769 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980771 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980774 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980777 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980779 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980782 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 03:55:17.987332 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980787 2576 flags.go:64] FLAG: --kube-reserved="" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980790 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980793 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980796 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980799 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980802 2576 flags.go:64] FLAG: --lock-file="" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980805 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980808 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980811 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980816 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980819 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980822 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980824 2576 flags.go:64] FLAG: --logging-format="text" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980827 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980830 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980836 2576 flags.go:64] FLAG: --manifest-url="" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980839 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980843 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980846 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980849 2576 flags.go:64] FLAG: --max-pods="110" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980852 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980855 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980858 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980861 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980863 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 03:55:17.987916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980866 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980869 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980889 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980892 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980895 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980898 2576 flags.go:64] FLAG: --pod-cidr="" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980901 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980907 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980910 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980913 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980915 2576 flags.go:64] FLAG: --port="10250" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980919 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980921 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c2a0c2acad0abc15" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980924 2576 flags.go:64] FLAG: --qos-reserved="" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980927 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980930 2576 flags.go:64] FLAG: --register-node="true" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980933 2576 flags.go:64] FLAG: --register-with-taints="" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980937 2576 flags.go:64] FLAG: --registry-burst="10" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980940 2576 flags.go:64] FLAG: --registry-qps="5" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980943 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980945 2576 flags.go:64] FLAG: --reserved-memory="" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980949 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980952 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980957 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980960 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 03:55:17.988520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980962 2576 flags.go:64] FLAG: --runonce="false" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980965 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980968 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980971 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980973 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980976 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980979 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980982 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980984 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980987 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980990 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980993 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980995 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.980998 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981002 2576 flags.go:64] FLAG: --system-cgroups="" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981005 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981010 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981012 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981015 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981018 2576 flags.go:64] FLAG: --tls-min-version="" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981021 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981024 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981026 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981029 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981032 2576 flags.go:64] FLAG: --v="2" Jul 29 03:55:17.989155 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981035 2576 flags.go:64] FLAG: --version="false" Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981039 2576 flags.go:64] FLAG: --vmodule="" Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981043 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.981046 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981154 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981159 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981162 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981165 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981168 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981171 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981175 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981179 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981182 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981185 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981187 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981190 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981193 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981195 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981198 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981201 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 29 03:55:17.989751 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981203 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981207 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981209 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981212 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981216 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981218 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981221 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981224 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981226 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981229 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981231 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981234 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981236 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981239 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981242 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981244 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981247 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981250 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981253 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981255 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 03:55:17.990241 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981258 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981260 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981263 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981265 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981269 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981272 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981275 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981278 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981281 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981283 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981286 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981289 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981291 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981295 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981297 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981300 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981302 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981305 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981307 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 03:55:17.990742 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981310 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981313 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981315 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981318 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981320 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981323 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981325 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981328 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981331 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981333 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981337 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981339 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981342 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981345 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981348 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981351 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981355 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981357 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981360 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981363 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 03:55:17.991223 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981366 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981368 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981371 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981373 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981375 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981378 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981381 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981384 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981387 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981389 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981392 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981394 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981396 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981399 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981401 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981404 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981407 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981409 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981411 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 03:55:17.991712 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981414 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981416 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981419 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981423 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.981426 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.982187 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.990988 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.991001 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.991013 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.991019 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991082 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991087 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991090 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991093 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991096 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 03:55:17.992180 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991099 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991101 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991104 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991106 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991109 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991112 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991114 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991117 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991119 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991122 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991125 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991127 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991130 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991132 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991136 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991140 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991143 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991146 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991148 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991157 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 03:55:17.992548 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991160 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991163 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991166 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991170 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991175 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991178 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991181 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991184 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991186 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991189 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991192 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991194 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991196 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991199 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991202 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991204 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991206 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991209 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991211 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 03:55:17.993058 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991214 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991216 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991219 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991221 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991223 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991226 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991229 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991231 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991235 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991237 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991240 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991242 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991245 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991253 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991255 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991258 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991260 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991263 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991266 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991268 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 03:55:17.993513 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991271 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991273 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991276 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991278 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991281 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991283 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991285 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991288 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991291 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991295 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991298 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991300 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991303 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991306 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991308 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991311 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991313 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991315 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991318 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991320 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 03:55:17.994119 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991323 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991325 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991327 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991330 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991332 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991335 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991342 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991345 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991347 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991350 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991352 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991355 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991358 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991360 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991362 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 03:55:17.994584 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.991367 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991511 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991515 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991518 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991520 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991523 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991526 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991528 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991530 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991533 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991535 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991538 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991540 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991542 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991545 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991553 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991556 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991558 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991561 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991563 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 03:55:17.994950 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991566 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991568 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991572 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991588 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991597 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991600 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991603 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991606 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991609 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991613 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991615 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991618 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991621 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991623 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991626 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991628 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991631 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991633 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991636 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991638 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 03:55:17.995415 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991640 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991644 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991647 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991650 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991653 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991655 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991658 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991661 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991664 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991666 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991669 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991671 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991674 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991676 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991679 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991681 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991684 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991691 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991694 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991697 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 03:55:17.995875 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991699 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991702 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991704 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991707 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991709 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991712 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991714 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991717 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991719 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991722 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991725 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991727 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991731 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991734 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991737 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991739 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991742 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991745 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991747 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 03:55:17.996361 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991750 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991752 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991755 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991757 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991759 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991762 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991764 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991766 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991769 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991771 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991774 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991782 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991784 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991787 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991790 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991792 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991794 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991797 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991799 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991802 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 03:55:17.996810 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:17.991804 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 03:55:17.997297 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.991808 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 03:55:17.997297 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.991918 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 03:55:17.997297 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.993852 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 03:55:17.997297 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.995212 2576 server.go:1019] "Starting client certificate rotation" Jul 29 03:55:17.997297 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.995308 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 03:55:17.997297 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.995344 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 03:55:17.998174 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:17.998151 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 03:55:18.002121 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.002102 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 03:55:18.015876 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.015859 2576 log.go:25] "Validated CRI v1 runtime API" Jul 29 03:55:18.017734 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.017721 2576 log.go:25] "Validated CRI v1 image API" Jul 29 03:55:18.019310 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.019290 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 03:55:18.021627 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.021609 2576 fs.go:135] Filesystem UUIDs: map[73e203b8-2a20-4797-af9a-900b089cc7b1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d3ce12de-ac17-4645-8f56-85c661e07f2d:/dev/nvme0n1p4] Jul 29 03:55:18.021698 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.021625 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 03:55:18.024230 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.024214 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 03:55:18.027717 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.027613 2576 manager.go:216] Machine: {Timestamp:2026-07-29 03:55:18.026389137 +0000 UTC m=+0.318485490 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102946 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec238d6702b2038d74d87d17f633d9a8 SystemUUID:ec238d67-02b2-038d-74d8-7d17f633d9a8 BootID:4fb7f6fd-5c52-40e7-b9d6-ab27b7a515f1 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7b:9d:dd:4b:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7b:9d:dd:4b:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:81:2c:b1:05:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 03:55:18.027717 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.027712 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 03:55:18.027812 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.027780 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 03:55:18.029607 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.029571 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 03:55:18.029759 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.029611 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-47.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 03:55:18.030388 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.030376 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 03:55:18.030443 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.030394 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 03:55:18.030443 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.030405 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 03:55:18.031204 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.031192 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 03:55:18.031977 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.031966 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 03:55:18.034819 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.034805 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 29 03:55:18.034972 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.034960 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 03:55:18.035031 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.035009 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 29 03:55:18.035089 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.035047 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 03:55:18.035089 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.035063 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 03:55:18.035089 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.035077 2576 kubelet.go:398] "Adding apiserver pod source" Jul 29 03:55:18.035204 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.035090 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 03:55:18.035447 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.035432 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 03:55:18.035507 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.035456 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 03:55:18.037548 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.037526 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 03:55:18.039658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.039642 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 03:55:18.039745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.039681 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 03:55:18.041114 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041100 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041122 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041132 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041141 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041153 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041163 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041174 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041182 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041192 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 03:55:18.041198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041200 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 03:55:18.041458 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041212 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 03:55:18.041458 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.041221 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 03:55:18.042727 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.042714 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 03:55:18.042780 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.042729 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 03:55:18.045743 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.045716 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 03:55:18.045743 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.045740 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-47.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 03:55:18.045908 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.045721 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 03:55:18.045908 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.045827 2576 server.go:1268] "Started kubelet" Jul 29 03:55:18.045986 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.045921 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 03:55:18.045986 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.045921 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 03:55:18.045986 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.045982 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 03:55:18.046682 ip-10-0-140-47 systemd[1]: Started Kubernetes Kubelet. Jul 29 03:55:18.047636 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.047504 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 03:55:18.052637 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.052611 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 03:55:18.052988 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.052968 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 29 03:55:18.055112 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.054308 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-47.ec2.internal.18c6a673ac93e48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-47.ec2.internal,UID:ip-10-0-140-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-47.ec2.internal,},FirstTimestamp:2026-07-29 03:55:18.045803659 +0000 UTC m=+0.337900013,LastTimestamp:2026-07-29 03:55:18.045803659 +0000 UTC m=+0.337900013,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-47.ec2.internal,}" Jul 29 03:55:18.058071 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058052 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 03:55:18.058071 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058065 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 03:55:18.058256 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.058165 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.058720 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058700 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 03:55:18.058798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058729 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 03:55:18.058798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058700 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 03:55:18.058906 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058842 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 29 03:55:18.058906 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058849 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 29 03:55:18.058906 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058857 2576 factory.go:55] Registering systemd factory Jul 29 03:55:18.058906 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.058872 2576 factory.go:223] Registration of the systemd container factory successfully Jul 29 03:55:18.059085 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.059074 2576 factory.go:153] Registering CRI-O factory Jul 29 03:55:18.059085 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.059085 2576 factory.go:223] Registration of the crio container factory successfully Jul 29 03:55:18.059174 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.059138 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 03:55:18.059174 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.059163 2576 factory.go:103] Registering Raw factory Jul 29 03:55:18.059278 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.059181 2576 manager.go:1196] Started watching for new ooms in manager Jul 29 03:55:18.059679 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.059667 2576 manager.go:349] Starting recovery of all containers Jul 29 03:55:18.063139 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.063117 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 03:55:18.063232 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.063211 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 03:55:18.070602 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.070450 2576 manager.go:354] Recovery completed Jul 29 03:55:18.071985 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.071963 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 29 03:55:18.074153 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.074133 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnrsf" Jul 29 03:55:18.075198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.075187 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 03:55:18.076996 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.076982 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientMemory" Jul 29 03:55:18.077069 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.077008 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 03:55:18.077069 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.077017 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientPID" Jul 29 03:55:18.077497 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.077481 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 03:55:18.077497 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.077494 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 03:55:18.077645 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.077494 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 03:55:18.077645 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.077556 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 29 03:55:18.078366 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.078351 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientMemory" Jul 29 03:55:18.078417 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.078376 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 03:55:18.078417 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.078397 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientPID" Jul 29 03:55:18.079576 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.079493 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-47.ec2.internal.18c6a673ae6fd785 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-47.ec2.internal,UID:ip-10-0-140-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-47.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-47.ec2.internal,},FirstTimestamp:2026-07-29 03:55:18.076995461 +0000 UTC m=+0.369091817,LastTimestamp:2026-07-29 03:55:18.076995461 +0000 UTC m=+0.369091817,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-47.ec2.internal,}" Jul 29 03:55:18.080158 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.080143 2576 policy_none.go:49] "None policy: Start" Jul 29 03:55:18.080158 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.080160 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 03:55:18.080247 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.080170 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 03:55:18.081286 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.081272 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnrsf" Jul 29 03:55:18.082524 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.082513 2576 policy_none.go:47] "Start" Jul 29 03:55:18.115418 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.115405 2576 manager.go:337] "Starting Device Plugin manager" Jul 29 03:55:18.115488 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.115439 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 03:55:18.115488 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.115449 2576 server.go:87] "Starting device plugin registration server" Jul 29 03:55:18.115709 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.115690 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 03:55:18.115790 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.115706 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 03:55:18.115835 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.115797 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 03:55:18.115875 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.115866 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 03:55:18.115875 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.115874 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 03:55:18.116383 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.116362 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 03:55:18.116452 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.116408 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.189248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.189224 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 03:55:18.191253 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.190434 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 03:55:18.191253 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.190463 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 03:55:18.191253 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.190481 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 03:55:18.191253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.190509 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 03:55:18.192365 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.192297 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 03:55:18.216424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.216405 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 03:55:18.217175 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.217159 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientMemory" Jul 29 03:55:18.217265 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.217192 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 03:55:18.217265 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.217218 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientPID" Jul 29 03:55:18.217265 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.217248 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.225911 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.225856 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.225911 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.225874 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-47.ec2.internal\": node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.248078 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.248058 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.290898 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.290856 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal"] Jul 29 03:55:18.290990 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.290954 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 03:55:18.292183 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.292163 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientMemory" Jul 29 03:55:18.292252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.292191 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 03:55:18.292252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.292201 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientPID" Jul 29 03:55:18.294378 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.294366 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 03:55:18.294502 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.294486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.294535 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.294515 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 03:55:18.295100 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.295085 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientMemory" Jul 29 03:55:18.295100 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.295093 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientMemory" Jul 29 03:55:18.295205 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.295119 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 03:55:18.295205 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.295120 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 03:55:18.295205 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.295135 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientPID" Jul 29 03:55:18.295205 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.295151 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientPID" Jul 29 03:55:18.297500 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.297483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.297542 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.297528 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 03:55:18.298214 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.298188 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientMemory" Jul 29 03:55:18.298214 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.298213 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 03:55:18.298299 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.298224 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeHasSufficientPID" Jul 29 03:55:18.337483 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.337459 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-47.ec2.internal\" not found" node="ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.341810 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.341792 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-47.ec2.internal\" not found" node="ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.348835 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.348821 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.360396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.360373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b753081a1bef3704e4855e1302d0c324-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.ec2.internal\" (UID: \"b753081a1bef3704e4855e1302d0c324\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.360475 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.360403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce28952636746a3e81d456712aaff32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal\" (UID: \"cce28952636746a3e81d456712aaff32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.360475 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.360421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce28952636746a3e81d456712aaff32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal\" (UID: \"cce28952636746a3e81d456712aaff32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.449685 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.449654 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.460965 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.460945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b753081a1bef3704e4855e1302d0c324-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.ec2.internal\" (UID: \"b753081a1bef3704e4855e1302d0c324\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.461008 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.460967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce28952636746a3e81d456712aaff32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal\" (UID: \"cce28952636746a3e81d456712aaff32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.461008 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.460985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce28952636746a3e81d456712aaff32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal\" (UID: \"cce28952636746a3e81d456712aaff32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.461084 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.461026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce28952636746a3e81d456712aaff32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal\" (UID: \"cce28952636746a3e81d456712aaff32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.461084 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.461043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b753081a1bef3704e4855e1302d0c324-config\") pod \"kube-apiserver-proxy-ip-10-0-140-47.ec2.internal\" (UID: \"b753081a1bef3704e4855e1302d0c324\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.461084 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.461050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce28952636746a3e81d456712aaff32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal\" (UID: \"cce28952636746a3e81d456712aaff32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.550432 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.550360 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.639931 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.639899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.644904 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.644626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:18.651279 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.651260 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.751861 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.751832 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.852361 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.852301 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.952873 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:18.952842 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:18.994319 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.994291 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 03:55:18.994838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:18.994457 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 03:55:19.053202 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:19.053175 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:19.086064 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.086014 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 03:50:18 +0000 UTC" deadline="2028-05-09 13:41:05.4098989 +0000 UTC" Jul 29 03:55:19.086064 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.086053 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15609h45m46.323850437s" Jul 29 03:55:19.154169 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:19.154138 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:19.239870 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:19.239822 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb753081a1bef3704e4855e1302d0c324.slice/crio-02914ba144988533b060c70f7b172450ebda0a7ad61eaa9ef8e045c45646230b WatchSource:0}: Error finding container 02914ba144988533b060c70f7b172450ebda0a7ad61eaa9ef8e045c45646230b: Status 404 returned error can't find the container with id 02914ba144988533b060c70f7b172450ebda0a7ad61eaa9ef8e045c45646230b Jul 29 03:55:19.246470 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.246452 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 03:55:19.252675 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:19.252648 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce28952636746a3e81d456712aaff32.slice/crio-8869d8733cee0cfa496c30ea9b44c81b366f051c96175738e9845e3af95af194 WatchSource:0}: Error finding container 8869d8733cee0cfa496c30ea9b44c81b366f051c96175738e9845e3af95af194: Status 404 returned error can't find the container with id 8869d8733cee0cfa496c30ea9b44c81b366f051c96175738e9845e3af95af194 Jul 29 03:55:19.254404 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:19.254387 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:19.356673 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:19.356639 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-47.ec2.internal\" not found" Jul 29 03:55:19.414856 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.414809 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 03:55:19.459161 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.459138 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" Jul 29 03:55:19.469972 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.469953 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 03:55:19.471833 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.471813 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" Jul 29 03:55:19.483493 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.483471 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 03:55:19.574859 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.574836 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 03:55:19.580657 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:19.580640 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 03:55:20.036245 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.036012 2576 apiserver.go:52] "Watching apiserver" Jul 29 03:55:20.045076 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.045032 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 03:55:20.047176 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.047147 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x6mqc","openshift-multus/multus-7mv9l","openshift-multus/multus-additional-cni-plugins-9gt4v","openshift-multus/network-metrics-daemon-qr5hw","openshift-network-operator/iptables-alerter-29plw","openshift-ovn-kubernetes/ovnkube-node-rxm72","kube-system/konnectivity-agent-6jmqc","kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz","openshift-cluster-node-tuning-operator/tuned-6slpm","openshift-dns/node-resolver-wd8bl","openshift-image-registry/node-ca-tnhv8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal","openshift-network-diagnostics/network-check-target-xznfq"] Jul 29 03:55:20.050252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.050231 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.054191 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.054166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.054778 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.054522 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 03:55:20.056429 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.055227 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 03:55:20.056429 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.055483 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zwbqx\"" Jul 29 03:55:20.056429 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.055767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.058126 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.057801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:20.058126 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.057879 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:20.058378 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.058354 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 03:55:20.059607 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.059580 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 03:55:20.059722 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.059651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-25jxj\"" Jul 29 03:55:20.059814 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.059791 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 03:55:20.059874 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.059830 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 03:55:20.060144 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.060101 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 03:55:20.060305 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.060287 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 03:55:20.060515 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.060490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pv7zv\"" Jul 29 03:55:20.060629 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.060546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.060695 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.060674 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 03:55:20.060883 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.060862 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 03:55:20.063087 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.063067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.064105 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.064082 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:55:20.064522 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.064501 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kffgr\"" Jul 29 03:55:20.064826 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.064807 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 03:55:20.064987 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.064968 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 03:55:20.065359 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.065340 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zbh6s\"" Jul 29 03:55:20.065638 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.065619 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 03:55:20.065736 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.065717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.066404 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.066381 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 03:55:20.066404 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.066392 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 03:55:20.066744 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.066724 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 03:55:20.067390 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.067370 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 03:55:20.067605 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.067558 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 03:55:20.069080 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.069062 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 03:55:20.069437 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.069419 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfc7m\"" Jul 29 03:55:20.069838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.069812 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 03:55:20.070288 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:20.070402 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.070335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:20.070569 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edca332e-79b6-46ec-87d2-0573afcbc050-host-slash\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.070644 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkl9\" (UniqueName: \"kubernetes.io/projected/edca332e-79b6-46ec-87d2-0573afcbc050-kube-api-access-9dkl9\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.070644 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12b19858-6ae7-4c40-b65a-209e52534efd-konnectivity-ca\") pod \"konnectivity-agent-6jmqc\" (UID: \"12b19858-6ae7-4c40-b65a-209e52534efd\") " pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.070644 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:20.070644 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wdf\" (UniqueName: \"kubernetes.io/projected/e976f8e0-d011-47d3-a2c4-61198b51f2b9-kube-api-access-s6wdf\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-tmp-dir\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edca332e-79b6-46ec-87d2-0573afcbc050-iptables-alerter-script\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpkd\" (UniqueName: \"kubernetes.io/projected/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-kube-api-access-8fpkd\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-os-release\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch57t\" (UniqueName: \"kubernetes.io/projected/4b079719-cf88-4c8b-98fe-0f72beec6c97-kube-api-access-ch57t\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12b19858-6ae7-4c40-b65a-209e52534efd-agent-certs\") pod \"konnectivity-agent-6jmqc\" (UID: \"12b19858-6ae7-4c40-b65a-209e52534efd\") " pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.070867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-hosts-file\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.071418 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-system-cni-dir\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.071418 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-cnibin\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.071418 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.070951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.072392 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.072332 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 03:55:20.072550 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.072516 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 03:55:20.073259 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.073227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.076082 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.076061 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x7ncc\"" Jul 29 03:55:20.076342 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.076322 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 03:55:20.076491 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.076473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.079201 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.079181 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:55:20.079446 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.079422 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bgncs\"" Jul 29 03:55:20.079446 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.079442 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 03:55:20.080643 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.080617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.080781 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.080753 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:20.082832 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.082812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.085628 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.085587 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lm8lv\"" Jul 29 03:55:20.085914 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.085887 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 03:55:20.086101 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.086084 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 03:55:20.086318 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.086301 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 03:55:20.099053 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.099035 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nbbdp" Jul 29 03:55:20.107109 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.107088 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nbbdp" Jul 29 03:55:20.160035 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.160015 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 03:55:20.161319 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.161301 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 03:55:20.171605 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.171577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-cnibin\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.171726 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.171644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edca332e-79b6-46ec-87d2-0573afcbc050-host-slash\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.171726 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.171676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkl9\" (UniqueName: \"kubernetes.io/projected/edca332e-79b6-46ec-87d2-0573afcbc050-kube-api-access-9dkl9\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.171726 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.171708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-cni-netd\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.171898 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.171726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-cnibin\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.171898 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.171734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysctl-conf\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.171898 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.171794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edca332e-79b6-46ec-87d2-0573afcbc050-host-slash\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.172136 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-os-release\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.172136 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d5643a1-ea0e-4024-9295-d295ca13a9f1-cni-binary-copy\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.172248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-netns\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.172248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:20.172248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.172408 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-modprobe-d\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.172408 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-lib-modules\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.172408 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysconfig\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.172408 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-cni-multus\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.172608 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-var-lib-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.172608 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.172461 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:20.172608 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-ovn\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.172608 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.172578 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs podName:e976f8e0-d011-47d3-a2c4-61198b51f2b9 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:20.672554786 +0000 UTC m=+2.964651165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs") pod "network-metrics-daemon-qr5hw" (UID: "e976f8e0-d011-47d3-a2c4-61198b51f2b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:20.172838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-cni-bin\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.172838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-dbus\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.172838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-hosts-file\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.172838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12b19858-6ae7-4c40-b65a-209e52534efd-konnectivity-ca\") pod \"konnectivity-agent-6jmqc\" (UID: \"12b19858-6ae7-4c40-b65a-209e52534efd\") " pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.172838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-run-netns\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.172838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/983f6173-e462-45e6-aa42-2e2ac529c591-etc-tuned\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.172838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-cni-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-hosts-file\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-kubelet\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.172995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-daemon-config\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-etc-kubernetes\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-env-overrides\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdfn\" (UniqueName: \"kubernetes.io/projected/e1f40247-25b6-483a-980b-2b869d2cbe97-kube-api-access-nvdfn\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-socket-dir-parent\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-serviceca\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.173252 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28n7r\" (UniqueName: \"kubernetes.io/projected/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-kube-api-access-28n7r\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edca332e-79b6-46ec-87d2-0573afcbc050-iptables-alerter-script\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1f40247-25b6-483a-980b-2b869d2cbe97-ovn-node-metrics-cert\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpkd\" (UniqueName: \"kubernetes.io/projected/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-kube-api-access-8fpkd\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12b19858-6ae7-4c40-b65a-209e52534efd-konnectivity-ca\") pod \"konnectivity-agent-6jmqc\" (UID: \"12b19858-6ae7-4c40-b65a-209e52534efd\") " pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12b19858-6ae7-4c40-b65a-209e52534efd-agent-certs\") pod \"konnectivity-agent-6jmqc\" (UID: \"12b19858-6ae7-4c40-b65a-209e52534efd\") " pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-node-log\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-device-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.173745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnmd\" (UniqueName: \"kubernetes.io/projected/4d5643a1-ea0e-4024-9295-d295ca13a9f1-kube-api-access-bmnmd\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.173827 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-etc-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983f6173-e462-45e6-aa42-2e2ac529c591-tmp\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-kubelet-config\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-host\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-etc-selinux\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-run\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edca332e-79b6-46ec-87d2-0573afcbc050-iptables-alerter-script\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-k8s-cni-cncf-io\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.174383 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-systemd-units\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-log-socket\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-socket-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-registration-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-cni-bin\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-multus-certs\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-sys-fs\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-hostroot\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-system-cni-dir\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-kubelet\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-systemd\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-system-cni-dir\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.174899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sgxn\" (UniqueName: \"kubernetes.io/projected/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-kube-api-access-5sgxn\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysctl-d\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.174981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-systemd\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wdf\" (UniqueName: \"kubernetes.io/projected/e976f8e0-d011-47d3-a2c4-61198b51f2b9-kube-api-access-s6wdf\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-ovnkube-config\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-ovnkube-script-lib\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-kubernetes\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-sys\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-var-lib-kubelet\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bdd\" (UniqueName: \"kubernetes.io/projected/983f6173-e462-45e6-aa42-2e2ac529c591-kube-api-access-82bdd\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-system-cni-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b079719-cf88-4c8b-98fe-0f72beec6c97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-tmp-dir\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.175738 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-conf-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-slash\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-os-release\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch57t\" (UniqueName: \"kubernetes.io/projected/4b079719-cf88-4c8b-98fe-0f72beec6c97-kube-api-access-ch57t\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-host\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b079719-cf88-4c8b-98fe-0f72beec6c97-os-release\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-cnibin\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.176531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.175716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-tmp-dir\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.178985 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.178957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12b19858-6ae7-4c40-b65a-209e52534efd-agent-certs\") pod \"konnectivity-agent-6jmqc\" (UID: \"12b19858-6ae7-4c40-b65a-209e52534efd\") " pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.181930 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.181908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkl9\" (UniqueName: \"kubernetes.io/projected/edca332e-79b6-46ec-87d2-0573afcbc050-kube-api-access-9dkl9\") pod \"iptables-alerter-29plw\" (UID: \"edca332e-79b6-46ec-87d2-0573afcbc050\") " pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.186179 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.186157 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpkd\" (UniqueName: \"kubernetes.io/projected/0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b-kube-api-access-8fpkd\") pod \"node-resolver-wd8bl\" (UID: \"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b\") " pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.186553 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.186529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch57t\" (UniqueName: \"kubernetes.io/projected/4b079719-cf88-4c8b-98fe-0f72beec6c97-kube-api-access-ch57t\") pod \"multus-additional-cni-plugins-9gt4v\" (UID: \"4b079719-cf88-4c8b-98fe-0f72beec6c97\") " pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.188651 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.188624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wdf\" (UniqueName: \"kubernetes.io/projected/e976f8e0-d011-47d3-a2c4-61198b51f2b9-kube-api-access-s6wdf\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:20.195985 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.195948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" event={"ID":"cce28952636746a3e81d456712aaff32","Type":"ContainerStarted","Data":"8869d8733cee0cfa496c30ea9b44c81b366f051c96175738e9845e3af95af194"} Jul 29 03:55:20.197181 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.197158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" event={"ID":"b753081a1bef3704e4855e1302d0c324","Type":"ContainerStarted","Data":"02914ba144988533b060c70f7b172450ebda0a7ad61eaa9ef8e045c45646230b"} Jul 29 03:55:20.276507 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-slash\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-host\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-cnibin\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-slash\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-cni-netd\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysctl-conf\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-os-release\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d5643a1-ea0e-4024-9295-d295ca13a9f1-cni-binary-copy\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.276658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-host\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-netns\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276697 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-cnibin\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-modprobe-d\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-lib-modules\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysconfig\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-cni-multus\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276825 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-os-release\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-var-lib-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-ovn\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-var-lib-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-cni-bin\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-cni-netd\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276949 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-cni-bin\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.276974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-dbus\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-run-netns\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/983f6173-e462-45e6-aa42-2e2ac529c591-etc-tuned\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-cni-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-kubelet\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-lib-modules\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-daemon-config\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-etc-kubernetes\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-env-overrides\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.278147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.277508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdfn\" (UniqueName: \"kubernetes.io/projected/e1f40247-25b6-483a-980b-2b869d2cbe97-kube-api-access-nvdfn\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.278937 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.278911 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d5643a1-ea0e-4024-9295-d295ca13a9f1-cni-binary-copy\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.279091 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-socket-dir-parent\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.279188 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-socket-dir-parent\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.279255 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279186 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysconfig\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.279255 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279242 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-ovn\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.279354 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-cni-multus\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.279354 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-netns\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.279354 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.279492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-serviceca\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.279492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28n7r\" (UniqueName: \"kubernetes.io/projected/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-kube-api-access-28n7r\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.279492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1f40247-25b6-483a-980b-2b869d2cbe97-ovn-node-metrics-cert\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.279492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-dbus\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.279492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-node-log\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.279731 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.279731 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-device-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.279731 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnmd\" (UniqueName: \"kubernetes.io/projected/4d5643a1-ea0e-4024-9295-d295ca13a9f1-kube-api-access-bmnmd\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.279731 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-etc-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.279946 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983f6173-e462-45e6-aa42-2e2ac529c591-tmp\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.279946 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-kubelet-config\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.279946 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-host\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.279946 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-etc-selinux\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.279946 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-run\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.279946 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-k8s-cni-cncf-io\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-serviceca\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-systemd-units\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-log-socket\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-socket-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-registration-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-modprobe-d\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-cni-bin\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-multus-certs\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.280220 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:20.280275 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-sys-fs\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.280750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-hostroot\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.280750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-kubelet\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.280750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-cni-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.280750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-systemd\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.280750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-kubelet\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.280750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgxn\" (UniqueName: \"kubernetes.io/projected/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-kube-api-access-5sgxn\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.281082 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysctl-d\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.281082 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-systemd\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.281082 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.280950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-ovnkube-config\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.281248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-ovnkube-script-lib\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.281248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-kubernetes\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.281248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-sys\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.281248 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-var-lib-kubelet\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.281455 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bdd\" (UniqueName: \"kubernetes.io/projected/983f6173-e462-45e6-aa42-2e2ac529c591-kube-api-access-82bdd\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.281455 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-system-cni-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.281455 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.281455 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.281455 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:20.281701 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-conf-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.281701 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.281610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-conf-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.282240 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.282215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d5643a1-ea0e-4024-9295-d295ca13a9f1-multus-daemon-config\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysctl-d\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283029 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-etc-kubernetes\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-systemd\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283111 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-k8s-cni-cncf-io\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-node-log\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/983f6173-e462-45e6-aa42-2e2ac529c591-etc-tuned\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283359 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-systemd-units\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.283424 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283409 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-log-socket\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.279491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-run-netns\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-socket-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-device-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283625 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-registration-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-var-lib-cni-bin\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-hostroot\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-kubelet\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.283798 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret podName:525a2c3e-cca1-435f-a3d6-7b94f465d5f5 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:20.783778177 +0000 UTC m=+3.075874526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret") pod "global-pull-secret-syncer-x6mqc" (UID: "525a2c3e-cca1-435f-a3d6-7b94f465d5f5") : object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-systemd\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-host-run-multus-certs\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-etc-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.283934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-sys-fs\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-run-openvswitch\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d5643a1-ea0e-4024-9295-d295ca13a9f1-system-cni-dir\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-sys\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-kubernetes\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.286065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-var-lib-kubelet\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-host\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-kubelet-config\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284501 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-etc-selinux\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-run\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/983f6173-e462-45e6-aa42-2e2ac529c591-etc-sysctl-conf\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.284889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1f40247-25b6-483a-980b-2b869d2cbe97-ovn-node-metrics-cert\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.285036 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1f40247-25b6-483a-980b-2b869d2cbe97-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.285705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-env-overrides\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.286717 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-ovnkube-config\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.286925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.286882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983f6173-e462-45e6-aa42-2e2ac529c591-tmp\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.287414 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.287025 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1f40247-25b6-483a-980b-2b869d2cbe97-ovnkube-script-lib\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.290429 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.290406 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28n7r\" (UniqueName: \"kubernetes.io/projected/e1d96163-4e1b-4d7f-ad07-55d7488acb5e-kube-api-access-28n7r\") pod \"node-ca-tnhv8\" (UID: \"e1d96163-4e1b-4d7f-ad07-55d7488acb5e\") " pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.291369 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.290745 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 03:55:20.291369 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.290769 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 03:55:20.291369 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.290782 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pzzwf for pod openshift-network-diagnostics/network-check-target-xznfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:20.291369 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.290862 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf podName:4d3974fa-a38b-499b-b42a-c9abc7b6f6af nodeName:}" failed. No retries permitted until 2026-07-29 03:55:20.790844185 +0000 UTC m=+3.082940534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pzzwf" (UniqueName: "kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf") pod "network-check-target-xznfq" (UID: "4d3974fa-a38b-499b-b42a-c9abc7b6f6af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:20.291369 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.291156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgxn\" (UniqueName: \"kubernetes.io/projected/32a3eca8-fd3e-4ad4-b85d-33cdbbba4114-kube-api-access-5sgxn\") pod \"aws-ebs-csi-driver-node-tq8zz\" (UID: \"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.292455 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.292434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdfn\" (UniqueName: \"kubernetes.io/projected/e1f40247-25b6-483a-980b-2b869d2cbe97-kube-api-access-nvdfn\") pod \"ovnkube-node-rxm72\" (UID: \"e1f40247-25b6-483a-980b-2b869d2cbe97\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.292559 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.292544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnmd\" (UniqueName: \"kubernetes.io/projected/4d5643a1-ea0e-4024-9295-d295ca13a9f1-kube-api-access-bmnmd\") pod \"multus-7mv9l\" (UID: \"4d5643a1-ea0e-4024-9295-d295ca13a9f1\") " pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.294089 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.294067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bdd\" (UniqueName: \"kubernetes.io/projected/983f6173-e462-45e6-aa42-2e2ac529c591-kube-api-access-82bdd\") pod \"tuned-6slpm\" (UID: \"983f6173-e462-45e6-aa42-2e2ac529c591\") " pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.366064 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.366034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:20.376055 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.376030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd8bl" Jul 29 03:55:20.386747 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.386727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" Jul 29 03:55:20.401761 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.401700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29plw" Jul 29 03:55:20.413179 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.413160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:20.426076 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.426058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" Jul 29 03:55:20.434987 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.434968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7mv9l" Jul 29 03:55:20.445008 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.444990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6slpm" Jul 29 03:55:20.451860 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.451844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnhv8" Jul 29 03:55:20.684585 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.684499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:20.684761 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.684640 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:20.684761 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.684713 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs podName:e976f8e0-d011-47d3-a2c4-61198b51f2b9 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:21.684693864 +0000 UTC m=+3.976790210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs") pod "network-metrics-daemon-qr5hw" (UID: "e976f8e0-d011-47d3-a2c4-61198b51f2b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:20.785080 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.785045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:20.785253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.785183 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:20.785331 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.785260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret podName:525a2c3e-cca1-435f-a3d6-7b94f465d5f5 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:21.785236794 +0000 UTC m=+4.077333143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret") pod "global-pull-secret-syncer-x6mqc" (UID: "525a2c3e-cca1-435f-a3d6-7b94f465d5f5") : object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:20.886102 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:20.886062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:20.886291 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.886263 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 03:55:20.886409 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.886296 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 03:55:20.886409 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.886310 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pzzwf for pod openshift-network-diagnostics/network-check-target-xznfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:20.886409 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:20.886371 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf podName:4d3974fa-a38b-499b-b42a-c9abc7b6f6af nodeName:}" failed. No retries permitted until 2026-07-29 03:55:21.886350811 +0000 UTC m=+4.178447156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzzwf" (UniqueName: "kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf") pod "network-check-target-xznfq" (UID: "4d3974fa-a38b-499b-b42a-c9abc7b6f6af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:21.111362 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:21.111272 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 03:50:20 +0000 UTC" deadline="2028-02-23 16:17:08.443408416 +0000 UTC" Jul 29 03:55:21.111362 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:21.111327 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13788h21m47.332086035s" Jul 29 03:55:21.191175 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:21.191142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:21.191326 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.191268 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:21.287454 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.287419 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a3eca8_fd3e_4ad4_b85d_33cdbbba4114.slice/crio-ea044abb5e2ee0553b526fb25624ba177f2752109c4892dcb861a9ad5bfa0df1 WatchSource:0}: Error finding container ea044abb5e2ee0553b526fb25624ba177f2752109c4892dcb861a9ad5bfa0df1: Status 404 returned error can't find the container with id ea044abb5e2ee0553b526fb25624ba177f2752109c4892dcb861a9ad5bfa0df1 Jul 29 03:55:21.289523 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.289207 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983f6173_e462_45e6_aa42_2e2ac529c591.slice/crio-8792d241333a6ccdb7cf29526752168405c78ba13246faa08b927d28aa12ee9e WatchSource:0}: Error finding container 8792d241333a6ccdb7cf29526752168405c78ba13246faa08b927d28aa12ee9e: Status 404 returned error can't find the container with id 8792d241333a6ccdb7cf29526752168405c78ba13246faa08b927d28aa12ee9e Jul 29 03:55:21.291198 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.291171 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d96163_4e1b_4d7f_ad07_55d7488acb5e.slice/crio-cca5c3c4ed1ad87fa26989298e36b8d6e77271250afc1529b0b120073983e61a WatchSource:0}: Error finding container cca5c3c4ed1ad87fa26989298e36b8d6e77271250afc1529b0b120073983e61a: Status 404 returned error can't find the container with id cca5c3c4ed1ad87fa26989298e36b8d6e77271250afc1529b0b120073983e61a Jul 29 03:55:21.316858 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.316835 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7006fc_a2b3_4ff8_8126_e6e46a26fa7b.slice/crio-37a05385c86f71276a161b895c7c06b958961cfba7ec3b7665942faf6d314b8e WatchSource:0}: Error finding container 37a05385c86f71276a161b895c7c06b958961cfba7ec3b7665942faf6d314b8e: Status 404 returned error can't find the container with id 37a05385c86f71276a161b895c7c06b958961cfba7ec3b7665942faf6d314b8e Jul 29 03:55:21.318321 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.318299 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b079719_cf88_4c8b_98fe_0f72beec6c97.slice/crio-6c46f41fb376a2fdece56ffee60cf9a0f4b4ce98d96e2d24b84e7551ac0fcb41 WatchSource:0}: Error finding container 6c46f41fb376a2fdece56ffee60cf9a0f4b4ce98d96e2d24b84e7551ac0fcb41: Status 404 returned error can't find the container with id 6c46f41fb376a2fdece56ffee60cf9a0f4b4ce98d96e2d24b84e7551ac0fcb41 Jul 29 03:55:21.319453 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.319414 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5643a1_ea0e_4024_9295_d295ca13a9f1.slice/crio-11bd1bce022405a37181b00a3d1902fe523873dde6cd113acb540319fa25f1a5 WatchSource:0}: Error finding container 11bd1bce022405a37181b00a3d1902fe523873dde6cd113acb540319fa25f1a5: Status 404 returned error can't find the container with id 11bd1bce022405a37181b00a3d1902fe523873dde6cd113acb540319fa25f1a5 Jul 29 03:55:21.320719 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.320696 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f40247_25b6_483a_980b_2b869d2cbe97.slice/crio-482cd5881790820c1b02ada0184ad7c8aed937c62613f2dbc25fc28f8a058400 WatchSource:0}: Error finding container 482cd5881790820c1b02ada0184ad7c8aed937c62613f2dbc25fc28f8a058400: Status 404 returned error can't find the container with id 482cd5881790820c1b02ada0184ad7c8aed937c62613f2dbc25fc28f8a058400 Jul 29 03:55:21.321805 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.321775 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedca332e_79b6_46ec_87d2_0573afcbc050.slice/crio-f83ca3f04e0331fcb8398542a9e281f13b60ab714d349fd0b5efd61bc63cca14 WatchSource:0}: Error finding container f83ca3f04e0331fcb8398542a9e281f13b60ab714d349fd0b5efd61bc63cca14: Status 404 returned error can't find the container with id f83ca3f04e0331fcb8398542a9e281f13b60ab714d349fd0b5efd61bc63cca14 Jul 29 03:55:21.323176 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:21.322637 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b19858_6ae7_4c40_b65a_209e52534efd.slice/crio-d8b672ec9c5fe1f22a8ea18b8c14c322612a04ed5a99e596e20c550b67c873bf WatchSource:0}: Error finding container d8b672ec9c5fe1f22a8ea18b8c14c322612a04ed5a99e596e20c550b67c873bf: Status 404 returned error can't find the container with id d8b672ec9c5fe1f22a8ea18b8c14c322612a04ed5a99e596e20c550b67c873bf Jul 29 03:55:21.693730 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:21.693443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:21.693872 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.693556 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:21.693872 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.693825 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs podName:e976f8e0-d011-47d3-a2c4-61198b51f2b9 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:23.693802236 +0000 UTC m=+5.985898578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs") pod "network-metrics-daemon-qr5hw" (UID: "e976f8e0-d011-47d3-a2c4-61198b51f2b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:21.794627 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:21.794578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:21.794918 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.794899 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:21.795062 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.795044 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret podName:525a2c3e-cca1-435f-a3d6-7b94f465d5f5 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:23.794981126 +0000 UTC m=+6.087077491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret") pod "global-pull-secret-syncer-x6mqc" (UID: "525a2c3e-cca1-435f-a3d6-7b94f465d5f5") : object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:21.895631 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:21.895584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:21.895829 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.895803 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 03:55:21.895913 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.895838 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 03:55:21.895913 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.895854 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pzzwf for pod openshift-network-diagnostics/network-check-target-xznfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:21.896062 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:21.895926 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf podName:4d3974fa-a38b-499b-b42a-c9abc7b6f6af nodeName:}" failed. No retries permitted until 2026-07-29 03:55:23.895906876 +0000 UTC m=+6.188003222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzzwf" (UniqueName: "kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf") pod "network-check-target-xznfq" (UID: "4d3974fa-a38b-499b-b42a-c9abc7b6f6af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:22.111933 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.111818 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 03:50:20 +0000 UTC" deadline="2028-01-11 20:38:46.624422479 +0000 UTC" Jul 29 03:55:22.111933 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.111859 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12760h43m24.512568133s" Jul 29 03:55:22.203912 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.203880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:22.204102 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:22.204045 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:22.204102 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.204095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:22.204280 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:22.204217 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:22.229781 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.229742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" event={"ID":"b753081a1bef3704e4855e1302d0c324","Type":"ContainerStarted","Data":"dd67bcfaf6f44db94fdd91c17e79ef5c8952f4d5ffc95c979e8672868cc9e229"} Jul 29 03:55:22.251103 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.250046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6jmqc" event={"ID":"12b19858-6ae7-4c40-b65a-209e52534efd","Type":"ContainerStarted","Data":"d8b672ec9c5fe1f22a8ea18b8c14c322612a04ed5a99e596e20c550b67c873bf"} Jul 29 03:55:22.259099 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.258944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"482cd5881790820c1b02ada0184ad7c8aed937c62613f2dbc25fc28f8a058400"} Jul 29 03:55:22.263981 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.263891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerStarted","Data":"6c46f41fb376a2fdece56ffee60cf9a0f4b4ce98d96e2d24b84e7551ac0fcb41"} Jul 29 03:55:22.269984 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.269632 2576 generic.go:356] "Generic (PLEG): container finished" podID="cce28952636746a3e81d456712aaff32" containerID="6ec20465e861974d52c1326d7edc2044c0afe8fcaa0e3c3db667b51c02405ec4" exitCode=0 Jul 29 03:55:22.269984 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.269720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" event={"ID":"cce28952636746a3e81d456712aaff32","Type":"ContainerDied","Data":"6ec20465e861974d52c1326d7edc2044c0afe8fcaa0e3c3db667b51c02405ec4"} Jul 29 03:55:22.273008 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.272982 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29plw" event={"ID":"edca332e-79b6-46ec-87d2-0573afcbc050","Type":"ContainerStarted","Data":"f83ca3f04e0331fcb8398542a9e281f13b60ab714d349fd0b5efd61bc63cca14"} Jul 29 03:55:22.285562 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.285537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7mv9l" event={"ID":"4d5643a1-ea0e-4024-9295-d295ca13a9f1","Type":"ContainerStarted","Data":"11bd1bce022405a37181b00a3d1902fe523873dde6cd113acb540319fa25f1a5"} Jul 29 03:55:22.288056 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.288005 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-47.ec2.internal" podStartSLOduration=3.287987329 podStartE2EDuration="3.287987329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:22.246575739 +0000 UTC m=+4.538672105" watchObservedRunningTime="2026-07-29 03:55:22.287987329 +0000 UTC m=+4.580083696" Jul 29 03:55:22.298676 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.298564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd8bl" event={"ID":"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b","Type":"ContainerStarted","Data":"37a05385c86f71276a161b895c7c06b958961cfba7ec3b7665942faf6d314b8e"} Jul 29 03:55:22.320182 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.320150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnhv8" event={"ID":"e1d96163-4e1b-4d7f-ad07-55d7488acb5e","Type":"ContainerStarted","Data":"cca5c3c4ed1ad87fa26989298e36b8d6e77271250afc1529b0b120073983e61a"} Jul 29 03:55:22.341181 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.341147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6slpm" event={"ID":"983f6173-e462-45e6-aa42-2e2ac529c591","Type":"ContainerStarted","Data":"8792d241333a6ccdb7cf29526752168405c78ba13246faa08b927d28aa12ee9e"} Jul 29 03:55:22.343364 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:22.343341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" event={"ID":"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114","Type":"ContainerStarted","Data":"ea044abb5e2ee0553b526fb25624ba177f2752109c4892dcb861a9ad5bfa0df1"} Jul 29 03:55:23.192804 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:23.191298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:23.192804 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.191428 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:23.375018 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:23.374977 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" event={"ID":"cce28952636746a3e81d456712aaff32","Type":"ContainerStarted","Data":"dc14d8a241135a3523e0212c522011939ade19ead86ef6c3f319ccab3ded6b3d"} Jul 29 03:55:23.711138 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:23.710455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:23.711138 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.710642 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:23.711138 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.710710 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs podName:e976f8e0-d011-47d3-a2c4-61198b51f2b9 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:27.710689085 +0000 UTC m=+10.002785428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs") pod "network-metrics-daemon-qr5hw" (UID: "e976f8e0-d011-47d3-a2c4-61198b51f2b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:23.811109 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:23.811064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:23.811284 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.811217 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:23.811284 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.811278 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret podName:525a2c3e-cca1-435f-a3d6-7b94f465d5f5 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:27.811260494 +0000 UTC m=+10.103356840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret") pod "global-pull-secret-syncer-x6mqc" (UID: "525a2c3e-cca1-435f-a3d6-7b94f465d5f5") : object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:23.912212 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:23.912176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:23.912398 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.912346 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 03:55:23.912398 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.912366 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 03:55:23.912398 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.912378 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pzzwf for pod openshift-network-diagnostics/network-check-target-xznfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:23.912563 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:23.912433 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf podName:4d3974fa-a38b-499b-b42a-c9abc7b6f6af nodeName:}" failed. No retries permitted until 2026-07-29 03:55:27.912416447 +0000 UTC m=+10.204512807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzzwf" (UniqueName: "kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf") pod "network-check-target-xznfq" (UID: "4d3974fa-a38b-499b-b42a-c9abc7b6f6af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:24.193437 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:24.193343 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:24.193933 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:24.193480 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:24.193933 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:24.193848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:24.194061 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:24.193956 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:25.191216 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:25.191184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:25.191381 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:25.191319 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:26.191357 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:26.191325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:26.191715 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:26.191502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:26.194015 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:26.193990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:26.194157 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:26.194137 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:27.191257 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:27.191220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:27.191431 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.191348 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:27.746090 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:27.746052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:27.746269 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.746240 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:27.746344 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.746322 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs podName:e976f8e0-d011-47d3-a2c4-61198b51f2b9 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:35.746299508 +0000 UTC m=+18.038395855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs") pod "network-metrics-daemon-qr5hw" (UID: "e976f8e0-d011-47d3-a2c4-61198b51f2b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:27.846739 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:27.846701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:27.846940 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.846903 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:27.847018 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.846968 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret podName:525a2c3e-cca1-435f-a3d6-7b94f465d5f5 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:35.846950698 +0000 UTC m=+18.139047038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret") pod "global-pull-secret-syncer-x6mqc" (UID: "525a2c3e-cca1-435f-a3d6-7b94f465d5f5") : object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:27.947231 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:27.947194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:27.947447 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.947415 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 03:55:27.947447 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.947446 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 03:55:27.947634 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.947460 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pzzwf for pod openshift-network-diagnostics/network-check-target-xznfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:27.947634 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:27.947524 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf podName:4d3974fa-a38b-499b-b42a-c9abc7b6f6af nodeName:}" failed. No retries permitted until 2026-07-29 03:55:35.947503418 +0000 UTC m=+18.239599774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzzwf" (UniqueName: "kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf") pod "network-check-target-xznfq" (UID: "4d3974fa-a38b-499b-b42a-c9abc7b6f6af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:28.191091 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:28.191009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:28.191262 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:28.191139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:28.191477 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:28.191459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:28.191816 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:28.191542 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:29.191725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:29.191287 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:29.191725 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:29.191409 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:30.195457 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:30.194860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:30.195457 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:30.195005 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:30.195457 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:30.195321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:30.195457 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:30.195408 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:31.190946 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:31.190910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:31.191108 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:31.191045 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:32.193742 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:32.193714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:32.194180 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:32.193719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:32.194180 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:32.193805 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:32.194180 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:32.193942 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:33.191093 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:33.191063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:33.191279 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:33.191169 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:34.191103 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:34.191072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:34.191556 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:34.191200 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:34.191556 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:34.191235 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:34.191556 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:34.191321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:35.190865 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:35.190830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:35.191165 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:35.190973 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:35.813596 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:35.813566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:35.813765 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:35.813713 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:35.813842 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:35.813782 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs podName:e976f8e0-d011-47d3-a2c4-61198b51f2b9 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:51.813761886 +0000 UTC m=+34.105858240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs") pod "network-metrics-daemon-qr5hw" (UID: "e976f8e0-d011-47d3-a2c4-61198b51f2b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:35.914462 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:35.914423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:35.914643 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:35.914570 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:35.914643 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:35.914625 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret podName:525a2c3e-cca1-435f-a3d6-7b94f465d5f5 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:51.914607781 +0000 UTC m=+34.206704120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret") pod "global-pull-secret-syncer-x6mqc" (UID: "525a2c3e-cca1-435f-a3d6-7b94f465d5f5") : object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:36.015406 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:36.015369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:36.015565 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:36.015517 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 03:55:36.015565 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:36.015534 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 03:55:36.015565 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:36.015543 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pzzwf for pod openshift-network-diagnostics/network-check-target-xznfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:36.015682 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:36.015589 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf podName:4d3974fa-a38b-499b-b42a-c9abc7b6f6af nodeName:}" failed. No retries permitted until 2026-07-29 03:55:52.015575817 +0000 UTC m=+34.307672157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzzwf" (UniqueName: "kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf") pod "network-check-target-xznfq" (UID: "4d3974fa-a38b-499b-b42a-c9abc7b6f6af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:36.190867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:36.190802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:36.191012 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:36.190813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:36.191012 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:36.190924 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:36.191096 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:36.191008 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:37.190796 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:37.190757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:37.191213 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:37.190902 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:38.191865 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:38.191834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:38.192312 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:38.191984 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:38.192312 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:38.192039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:38.192312 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:38.192161 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:39.190827 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:39.190797 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:39.191023 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:39.190918 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:40.191104 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.190960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:40.191104 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:40.191077 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:40.191492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.191128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:40.191492 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:40.191232 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:40.414181 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.413976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6jmqc" event={"ID":"12b19858-6ae7-4c40-b65a-209e52534efd","Type":"ContainerStarted","Data":"24df38d9e8da4e52fdebf40edc601402a54216a1a17177c530950307129ccd6a"} Jul 29 03:55:40.415717 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.415383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerStarted","Data":"60e6af49e1f1a17f08478d218c62fef49f2a5c54342f491d41f2f4f99cd69db2"} Jul 29 03:55:40.420073 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.419954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd8bl" event={"ID":"0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b","Type":"ContainerStarted","Data":"65d526b6866b2401d25c82a27908851b97b31963d9a1a3d0b6d13745d4c9b97b"} Jul 29 03:55:40.421478 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.421459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" event={"ID":"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114","Type":"ContainerStarted","Data":"fcd5a6c2991e926123d41eb430a3ce8319367f9a93cff5e71d1dc58ffcfab4a0"} Jul 29 03:55:40.431040 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.431004 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6jmqc" podStartSLOduration=7.4621628 podStartE2EDuration="22.430993184s" totalImagesPullingTime="14.968830384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:40.430487971 +0000 UTC m=+22.722584335" watchObservedRunningTime="2026-07-29 03:55:40.430993184 +0000 UTC m=+22.723089545" Jul 29 03:55:40.431303 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.431278 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-47.ec2.internal" podStartSLOduration=21.431269778 podStartE2EDuration="21.431269778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:23.393111482 +0000 UTC m=+5.685207849" watchObservedRunningTime="2026-07-29 03:55:40.431269778 +0000 UTC m=+22.723366139" Jul 29 03:55:40.445652 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:40.445614 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wd8bl" podStartSLOduration=3.575774504 podStartE2EDuration="22.445603587s" totalImagesPullingTime="18.869829083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:40.445072797 +0000 UTC m=+22.737169162" watchObservedRunningTime="2026-07-29 03:55:40.445603587 +0000 UTC m=+22.737699948" Jul 29 03:55:41.191183 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.191006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:41.191815 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:41.191276 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:41.427502 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.427470 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnhv8" event={"ID":"e1d96163-4e1b-4d7f-ad07-55d7488acb5e","Type":"ContainerStarted","Data":"5fe1866e63b2848e0497dea272c5f7ca6910fa02070760d9fd18348cdb413958"} Jul 29 03:55:41.428980 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.428954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6slpm" event={"ID":"983f6173-e462-45e6-aa42-2e2ac529c591","Type":"ContainerStarted","Data":"1ce0704b0202102f9d24613b5795ad82281e40da3ce9a283223c1c85e7cd7e09"} Jul 29 03:55:41.432907 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.432867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"ff3bf4882273751e61540c33ffc515fad97f5d4787482d6614a17e06d44698b5"} Jul 29 03:55:41.433003 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.432958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"e035cf9c71614b762e8b3f6a198eed732adc5f44fdbe90c529a6aa72c195fe85"} Jul 29 03:55:41.433003 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.432994 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"33de50c39ce40e5606062fbb9f0731ef6939f4adacda33027abe39b1e15612f6"} Jul 29 03:55:41.433111 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.433035 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"04e818e72cac5b8aecb3bc87e2f695e079a565450eaf4bc50d711f0dad565c81"} Jul 29 03:55:41.433111 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.433097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"4c1ea41eab9d9bebad1324845cc82181d342cb37dbb00cbe258c7ccb380587d3"} Jul 29 03:55:41.433201 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.433129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"4354d5ea177655261ab0229a60e9a0f58119273700ca6cc7219c38594fa309b4"} Jul 29 03:55:41.434286 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.434264 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b079719-cf88-4c8b-98fe-0f72beec6c97" containerID="60e6af49e1f1a17f08478d218c62fef49f2a5c54342f491d41f2f4f99cd69db2" exitCode=0 Jul 29 03:55:41.434377 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.434346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerDied","Data":"60e6af49e1f1a17f08478d218c62fef49f2a5c54342f491d41f2f4f99cd69db2"} Jul 29 03:55:41.435808 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.435783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7mv9l" event={"ID":"4d5643a1-ea0e-4024-9295-d295ca13a9f1","Type":"ContainerStarted","Data":"9fd727339b7bc929d1bf335203fcb052363f58c371e2c57bfa72da8c43e3774f"} Jul 29 03:55:41.443324 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.443254 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tnhv8" podStartSLOduration=4.5696320440000004 podStartE2EDuration="23.443241434s" totalImagesPullingTime="18.87360939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:41.443107921 +0000 UTC m=+23.735204295" watchObservedRunningTime="2026-07-29 03:55:41.443241434 +0000 UTC m=+23.735337795" Jul 29 03:55:41.487307 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.487271 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7mv9l" podStartSLOduration=4.41365176 podStartE2EDuration="23.487257354s" totalImagesPullingTime="19.073605594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:41.487145023 +0000 UTC m=+23.779241387" watchObservedRunningTime="2026-07-29 03:55:41.487257354 +0000 UTC m=+23.779353717" Jul 29 03:55:41.505602 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.505570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6slpm" podStartSLOduration=4.630224622 podStartE2EDuration="23.505557539s" totalImagesPullingTime="18.875332917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:41.505170248 +0000 UTC m=+23.797266611" watchObservedRunningTime="2026-07-29 03:55:41.505557539 +0000 UTC m=+23.797653925" Jul 29 03:55:41.533049 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:41.533026 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 03:55:42.135001 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.134907 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T03:55:41.533041902Z","UUID":"256e7ccb-2939-492c-b236-c0b29ebbe903","Handler":null,"Name":"","Endpoint":""} Jul 29 03:55:42.136935 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.136874 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 03:55:42.137077 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.136945 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 03:55:42.190928 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.190864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:42.191062 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.190866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:42.191062 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:42.191022 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:42.191189 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:42.191083 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:42.440193 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.440115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29plw" event={"ID":"edca332e-79b6-46ec-87d2-0573afcbc050","Type":"ContainerStarted","Data":"76898175a0634e36f630fcd62720ab69ade6380bf180e319c565e6af01c47c72"} Jul 29 03:55:42.442854 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.442807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" event={"ID":"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114","Type":"ContainerStarted","Data":"55eb9f8bff16c461e293384fcd2aa7cecc80f27d130e56b6673cc222a7f1e4fc"} Jul 29 03:55:42.457090 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:42.457041 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-29plw" podStartSLOduration=5.591739559 podStartE2EDuration="24.457022121s" totalImagesPullingTime="18.865282562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:42.456625976 +0000 UTC m=+24.748722337" watchObservedRunningTime="2026-07-29 03:55:42.457022121 +0000 UTC m=+24.749118464" Jul 29 03:55:43.191021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:43.190936 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:43.191172 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:43.191045 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:43.447843 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:43.447769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" event={"ID":"32a3eca8-fd3e-4ad4-b85d-33cdbbba4114","Type":"ContainerStarted","Data":"6ae013a30bcd6c491d7bcb6886dbd85c5e7bb137c57a8e9f0cfde1a96014b8bc"} Jul 29 03:55:43.452370 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:43.452336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"6d8d16817e6424f60541b871cb121cf2eea7167156313a7d72d494dfee20b046"} Jul 29 03:55:43.466404 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:43.466341 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tq8zz" podStartSLOduration=4.446971049 podStartE2EDuration="25.466322583s" totalImagesPullingTime="21.019351534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:43.465503823 +0000 UTC m=+25.757600185" watchObservedRunningTime="2026-07-29 03:55:43.466322583 +0000 UTC m=+25.758418948" Jul 29 03:55:44.191929 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:44.191689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:44.192105 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:44.191742 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:44.192105 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:44.192058 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:44.192211 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:44.192098 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:44.624499 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:44.624429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:44.625246 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:44.625226 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:45.191187 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:45.191158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:45.191387 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:45.191261 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:45.457080 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:45.457011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:45.457509 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:45.457483 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6jmqc" Jul 29 03:55:46.191876 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.191656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:46.192574 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.191685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:46.192574 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:46.192015 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:46.192574 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:46.192074 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:46.462677 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.462567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" event={"ID":"e1f40247-25b6-483a-980b-2b869d2cbe97","Type":"ContainerStarted","Data":"42bdab7472474f5a233345d59529c3f93b63bfbe05b3cc5c78c7cc47d0515594"} Jul 29 03:55:46.462867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.462848 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:46.464522 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.464498 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b079719-cf88-4c8b-98fe-0f72beec6c97" containerID="2c40c0835469e652c28dae2e0a69a63ba425c99cb149d283be7bbc34bd3c9b5e" exitCode=0 Jul 29 03:55:46.464639 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.464573 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerDied","Data":"2c40c0835469e652c28dae2e0a69a63ba425c99cb149d283be7bbc34bd3c9b5e"} Jul 29 03:55:46.482486 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.480418 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:46.488919 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:46.488863 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" podStartSLOduration=9.464807036 podStartE2EDuration="28.488851918s" totalImagesPullingTime="19.024044882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:46.487227238 +0000 UTC m=+28.779323599" watchObservedRunningTime="2026-07-29 03:55:46.488851918 +0000 UTC m=+28.780948280" Jul 29 03:55:47.191207 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.191179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:47.191497 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:47.191293 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:47.466780 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.466718 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:47.466780 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.466779 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:47.480896 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.480858 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:55:47.868907 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.868813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xznfq"] Jul 29 03:55:47.869070 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.868926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:47.869070 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:47.869032 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:47.871512 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.871488 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6mqc"] Jul 29 03:55:47.871625 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.871588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:47.871729 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:47.871692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:47.872077 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.872052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qr5hw"] Jul 29 03:55:47.872170 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:47.872157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:47.872281 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:47.872259 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:48.470399 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:48.470366 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b079719-cf88-4c8b-98fe-0f72beec6c97" containerID="bb0f875ca0a1b8bc679d5a8cd252a9dd7d12afb8d45deec29045024cdb3dbfa3" exitCode=0 Jul 29 03:55:48.470867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:48.470456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerDied","Data":"bb0f875ca0a1b8bc679d5a8cd252a9dd7d12afb8d45deec29045024cdb3dbfa3"} Jul 29 03:55:49.190665 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:49.190638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:49.190810 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:49.190749 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:49.190810 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:49.190802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:49.190898 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:49.190849 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:49.190977 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:49.190955 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:49.191043 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:49.191025 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:50.477087 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:50.477055 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b079719-cf88-4c8b-98fe-0f72beec6c97" containerID="bc8242ab449849d350b1de0bed375e23b1e76ce91b713267588421b01f0500bf" exitCode=0 Jul 29 03:55:50.477419 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:50.477121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerDied","Data":"bc8242ab449849d350b1de0bed375e23b1e76ce91b713267588421b01f0500bf"} Jul 29 03:55:51.191135 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:51.190945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:51.191307 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:51.191146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:51.191307 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:51.191237 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:51.191644 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:51.191614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:51.191725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:51.191680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:51.191788 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:51.191731 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:51.836232 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:51.836195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:51.836638 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:51.836328 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:51.836638 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:51.836388 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs podName:e976f8e0-d011-47d3-a2c4-61198b51f2b9 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:23.836370145 +0000 UTC m=+66.128466488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs") pod "network-metrics-daemon-qr5hw" (UID: "e976f8e0-d011-47d3-a2c4-61198b51f2b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 03:55:51.937313 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:51.937279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:51.937454 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:51.937404 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:51.937519 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:51.937463 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret podName:525a2c3e-cca1-435f-a3d6-7b94f465d5f5 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:23.937444024 +0000 UTC m=+66.229540366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret") pod "global-pull-secret-syncer-x6mqc" (UID: "525a2c3e-cca1-435f-a3d6-7b94f465d5f5") : object "kube-system"/"original-pull-secret" not registered Jul 29 03:55:52.038582 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:52.038545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:52.038761 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:52.038723 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 03:55:52.038761 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:52.038748 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 03:55:52.038761 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:52.038761 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pzzwf for pod openshift-network-diagnostics/network-check-target-xznfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:52.038892 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:52.038823 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf podName:4d3974fa-a38b-499b-b42a-c9abc7b6f6af nodeName:}" failed. No retries permitted until 2026-07-29 03:56:24.038803642 +0000 UTC m=+66.330899984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzzwf" (UniqueName: "kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf") pod "network-check-target-xznfq" (UID: "4d3974fa-a38b-499b-b42a-c9abc7b6f6af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 03:55:53.191108 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:53.190928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:53.191108 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:53.190965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:53.191108 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:53.190929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:53.191108 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:53.191063 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qr5hw" podUID="e976f8e0-d011-47d3-a2c4-61198b51f2b9" Jul 29 03:55:53.191739 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:53.191170 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xznfq" podUID="4d3974fa-a38b-499b-b42a-c9abc7b6f6af" Jul 29 03:55:53.191739 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:53.191260 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6mqc" podUID="525a2c3e-cca1-435f-a3d6-7b94f465d5f5" Jul 29 03:55:54.062143 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.062063 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-47.ec2.internal" event="NodeReady" Jul 29 03:55:54.062290 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.062208 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 03:55:54.095957 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.095927 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rhf5t"] Jul 29 03:55:54.099376 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.098633 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb"] Jul 29 03:55:54.099376 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.098981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.100219 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.100196 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw"] Jul 29 03:55:54.100526 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.100501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" Jul 29 03:55:54.101715 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.101694 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.101967 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.101948 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 29 03:55:54.101967 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.101963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.102251 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.102236 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 29 03:55:54.102507 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.102490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-69l77\"" Jul 29 03:55:54.102654 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.102641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.103243 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.103228 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.103761 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.103739 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65679c454d-bn24p"] Jul 29 03:55:54.104070 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.104052 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9kdqb\"" Jul 29 03:55:54.104070 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.104066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.104480 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.104464 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 03:55:54.104571 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.104547 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 03:55:54.105133 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.105109 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mbrgc\"" Jul 29 03:55:54.105133 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.105130 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.105290 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.105134 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.106244 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.105741 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl"] Jul 29 03:55:54.106244 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.105821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.107972 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.107953 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5"] Jul 29 03:55:54.108434 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.108399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 29 03:55:54.109207 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.108913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.110005 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.109987 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wjq2b\"" Jul 29 03:55:54.110093 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.110027 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 03:55:54.110153 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.110124 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 03:55:54.110153 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.110137 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-kqvg8"] Jul 29 03:55:54.110285 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.110265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:54.111017 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.110771 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 03:55:54.111928 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.111872 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.113101 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.112915 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 29 03:55:54.113101 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.112951 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 29 03:55:54.113101 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.112994 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.113101 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.112998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.113101 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.113027 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-544ng\"" Jul 29 03:55:54.113365 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.113157 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 03:55:54.113775 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.113756 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.113924 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.113870 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hj7p6\"" Jul 29 03:55:54.114027 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.113978 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.114386 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.114369 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 29 03:55:54.114655 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.114637 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qzqw6\"" Jul 29 03:55:54.114832 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.114813 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 29 03:55:54.115046 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.115033 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.115114 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.115065 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.115172 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.115145 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rhf5t"] Jul 29 03:55:54.119439 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.117987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb"] Jul 29 03:55:54.126328 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.126306 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 03:55:54.126964 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.126942 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl"] Jul 29 03:55:54.127531 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.127516 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n2lxw"] Jul 29 03:55:54.128519 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.128499 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 29 03:55:54.129373 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.129358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw"] Jul 29 03:55:54.129503 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.129489 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc"] Jul 29 03:55:54.129708 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.129507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.132351 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.132331 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 03:55:54.132552 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.132531 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhgfp\"" Jul 29 03:55:54.132735 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.132718 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 03:55:54.132961 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.132944 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77f89b64c-jk642"] Jul 29 03:55:54.133312 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.133293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.134706 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.134672 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kmqfs"] Jul 29 03:55:54.134935 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.134917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.136587 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.136568 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 29 03:55:54.136876 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.136855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.137068 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.136930 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 29 03:55:54.137147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.137137 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gdxwb\"" Jul 29 03:55:54.137294 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.137270 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.137421 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.137401 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8tqn7\"" Jul 29 03:55:54.137838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.137568 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 03:55:54.137838 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.137658 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.138465 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.138447 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27"] Jul 29 03:55:54.141471 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.139229 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 03:55:54.141764 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.139276 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.142030 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.139320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 03:55:54.142589 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.139494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.143773 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.143754 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 03:55:54.145479 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.145034 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.145479 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.145282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6rjhj\"" Jul 29 03:55:54.145479 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.145342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 03:55:54.145680 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.145551 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.145876 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.145859 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs"] Jul 29 03:55:54.147089 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.147073 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" Jul 29 03:55:54.148398 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.148381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-kqvg8"] Jul 29 03:55:54.148542 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.148528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5"] Jul 29 03:55:54.148752 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.148736 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77f89b64c-jk642"] Jul 29 03:55:54.148841 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.148767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc"] Jul 29 03:55:54.148841 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.148790 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27"] Jul 29 03:55:54.148841 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.148469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:54.149566 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.149550 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65679c454d-bn24p"] Jul 29 03:55:54.149676 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.149657 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5v647\"" Jul 29 03:55:54.149744 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.149633 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 03:55:54.150075 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.150060 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2lxw"] Jul 29 03:55:54.151782 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.151058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmqfs"] Jul 29 03:55:54.151782 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.151329 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 03:55:54.152948 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.152342 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 03:55:54.152948 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.152430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 03:55:54.152948 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.152614 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nxb4k\"" Jul 29 03:55:54.155220 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.155200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs"] Jul 29 03:55:54.255549 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-installation-pull-secrets\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bls5j\" (UniqueName: \"kubernetes.io/projected/96e39af8-6a4c-42d1-87d0-850180f27c86-kube-api-access-bls5j\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-registry-certificates\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpfn\" (UniqueName: \"kubernetes.io/projected/fd9e5cff-51ba-49af-9156-5d6714453eed-kube-api-access-rcpfn\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-bound-sa-token\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plh5n\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-kube-api-access-plh5n\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9e5cff-51ba-49af-9156-5d6714453eed-config-volume\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fae15b77-656e-4080-a60b-ced44902b861-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9h8\" (UniqueName: \"kubernetes.io/projected/fae15b77-656e-4080-a60b-ced44902b861-kube-api-access-wb9h8\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-image-registry-private-configuration\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.255989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.255982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-trusted-ca\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-tmp\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqtfp\" (UniqueName: \"kubernetes.io/projected/fec72562-75b7-41bc-8a75-86fd188e4023-kube-api-access-cqtfp\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/596212fb-ef59-4d1a-9770-86fce25ea36b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-default-certificate\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-serving-cert\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk627\" (UniqueName: \"kubernetes.io/projected/596212fb-ef59-4d1a-9770-86fce25ea36b-kube-api-access-vk627\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2sl\" (UniqueName: \"kubernetes.io/projected/db6c8ff1-edf0-406d-8438-e502e0a656c7-kube-api-access-nf2sl\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwqj\" (UniqueName: \"kubernetes.io/projected/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-kube-api-access-4zwqj\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f49df069-1566-41a1-9349-e85601dd1ffa-ca-trust-extracted\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db6c8ff1-edf0-406d-8438-e502e0a656c7-serving-cert\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.256467 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4cl\" (UniqueName: \"kubernetes.io/projected/525d31bf-6341-4d2d-b238-3313395d1e28-kube-api-access-jk4cl\") pod \"volume-data-source-validator-549c6d54cf-mc8zb\" (UID: \"525d31bf-6341-4d2d-b238-3313395d1e28\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96e39af8-6a4c-42d1-87d0-850180f27c86-config\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-stats-auth\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6sh\" (UniqueName: \"kubernetes.io/projected/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-kube-api-access-5z6sh\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbnd\" (UniqueName: \"kubernetes.io/projected/bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205-kube-api-access-rnbnd\") pod \"ingress-canary-kmqfs\" (UID: \"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205\") " pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96e39af8-6a4c-42d1-87d0-850180f27c86-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db6c8ff1-edf0-406d-8438-e502e0a656c7-trusted-ca\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9e5cff-51ba-49af-9156-5d6714453eed-tmp-dir\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bq9\" (UniqueName: \"kubernetes.io/projected/34fa7a1c-4bb6-4cef-9b34-105c75ceb894-kube-api-access-t9bq9\") pod \"network-check-source-7d88cb55d8-tfx27\" (UID: \"34fa7a1c-4bb6-4cef-9b34-105c75ceb894\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-snapshots\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/596212fb-ef59-4d1a-9770-86fce25ea36b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db6c8ff1-edf0-406d-8438-e502e0a656c7-config\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.257021 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-service-ca-bundle\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.257675 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.256992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a719ded6-afb9-4ac3-8c4f-1fea94627815-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:54.257675 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.257012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205-cert\") pod \"ingress-canary-kmqfs\" (UID: \"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205\") " pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.257675 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.257054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.267172 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.267126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:54.345098 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.345007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:54.357940 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.357912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plh5n\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-kube-api-access-plh5n\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.358095 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.357946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9e5cff-51ba-49af-9156-5d6714453eed-config-volume\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.358095 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.357975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fae15b77-656e-4080-a60b-ced44902b861-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.358095 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9h8\" (UniqueName: \"kubernetes.io/projected/fae15b77-656e-4080-a60b-ced44902b861-kube-api-access-wb9h8\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.358095 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-image-registry-private-configuration\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.358095 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-trusted-ca\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.358095 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-tmp\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqtfp\" (UniqueName: \"kubernetes.io/projected/fec72562-75b7-41bc-8a75-86fd188e4023-kube-api-access-cqtfp\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/596212fb-ef59-4d1a-9770-86fce25ea36b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-default-certificate\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-serving-cert\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk627\" (UniqueName: \"kubernetes.io/projected/596212fb-ef59-4d1a-9770-86fce25ea36b-kube-api-access-vk627\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2sl\" (UniqueName: \"kubernetes.io/projected/db6c8ff1-edf0-406d-8438-e502e0a656c7-kube-api-access-nf2sl\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.358396 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwqj\" (UniqueName: \"kubernetes.io/projected/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-kube-api-access-4zwqj\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f49df069-1566-41a1-9349-e85601dd1ffa-ca-trust-extracted\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db6c8ff1-edf0-406d-8438-e502e0a656c7-serving-cert\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4cl\" (UniqueName: \"kubernetes.io/projected/525d31bf-6341-4d2d-b238-3313395d1e28-kube-api-access-jk4cl\") pod \"volume-data-source-validator-549c6d54cf-mc8zb\" (UID: \"525d31bf-6341-4d2d-b238-3313395d1e28\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96e39af8-6a4c-42d1-87d0-850180f27c86-config\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9e5cff-51ba-49af-9156-5d6714453eed-config-volume\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-stats-auth\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-tmp\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.358712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fae15b77-656e-4080-a60b-ced44902b861-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.358791 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 03:55:54.358798 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.358804 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65679c454d-bn24p: secret "image-registry-tls" not found Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.358866 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls podName:f49df069-1566-41a1-9349-e85601dd1ffa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:54.858848803 +0000 UTC m=+37.150945150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls") pod "image-registry-65679c454d-bn24p" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa") : secret "image-registry-tls" not found Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6sh\" (UniqueName: \"kubernetes.io/projected/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-kube-api-access-5z6sh\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbnd\" (UniqueName: \"kubernetes.io/projected/bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205-kube-api-access-rnbnd\") pod \"ingress-canary-kmqfs\" (UID: \"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205\") " pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96e39af8-6a4c-42d1-87d0-850180f27c86-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f49df069-1566-41a1-9349-e85601dd1ffa-ca-trust-extracted\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db6c8ff1-edf0-406d-8438-e502e0a656c7-trusted-ca\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9e5cff-51ba-49af-9156-5d6714453eed-tmp-dir\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bq9\" (UniqueName: \"kubernetes.io/projected/34fa7a1c-4bb6-4cef-9b34-105c75ceb894-kube-api-access-t9bq9\") pod \"network-check-source-7d88cb55d8-tfx27\" (UID: \"34fa7a1c-4bb6-4cef-9b34-105c75ceb894\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-snapshots\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/596212fb-ef59-4d1a-9770-86fce25ea36b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db6c8ff1-edf0-406d-8438-e502e0a656c7-config\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.359413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-service-ca-bundle\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a719ded6-afb9-4ac3-8c4f-1fea94627815-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205-cert\") pod \"ingress-canary-kmqfs\" (UID: \"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205\") " pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-trusted-ca\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-installation-pull-secrets\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.359583 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:54.859566351 +0000 UTC m=+37.151662706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : configmap references non-existent config key: service-ca.crt Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.359618 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bls5j\" (UniqueName: \"kubernetes.io/projected/96e39af8-6a4c-42d1-87d0-850180f27c86-kube-api-access-bls5j\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-registry-certificates\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.359671 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:54.859655807 +0000 UTC m=+37.151752169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : secret "router-metrics-certs-default" not found Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpfn\" (UniqueName: \"kubernetes.io/projected/fd9e5cff-51ba-49af-9156-5d6714453eed-kube-api-access-rcpfn\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-bound-sa-token\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.360134 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.359984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db6c8ff1-edf0-406d-8438-e502e0a656c7-trusted-ca\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.360872 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.360114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-registry-certificates\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.360872 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.360205 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 03:55:54.360872 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.360244 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls podName:fd9e5cff-51ba-49af-9156-5d6714453eed nodeName:}" failed. No retries permitted until 2026-07-29 03:55:54.860231037 +0000 UTC m=+37.152327383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls") pod "dns-default-n2lxw" (UID: "fd9e5cff-51ba-49af-9156-5d6714453eed") : secret "dns-default-metrics-tls" not found Jul 29 03:55:54.360872 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.360441 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 03:55:54.360872 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.360496 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls podName:fec72562-75b7-41bc-8a75-86fd188e4023 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:54.860481402 +0000 UTC m=+37.152577744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls") pod "cluster-samples-operator-6748489656-qkfv5" (UID: "fec72562-75b7-41bc-8a75-86fd188e4023") : secret "samples-operator-tls" not found Jul 29 03:55:54.360872 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.360727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.360872 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.360728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-snapshots\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.361276 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.361069 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 03:55:54.361276 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.361120 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert podName:a719ded6-afb9-4ac3-8c4f-1fea94627815 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:54.861104974 +0000 UTC m=+37.153201316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bj6qs" (UID: "a719ded6-afb9-4ac3-8c4f-1fea94627815") : secret "networking-console-plugin-cert" not found Jul 29 03:55:54.361999 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.361974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96e39af8-6a4c-42d1-87d0-850180f27c86-config\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.362113 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.362076 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:54.362173 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.362132 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls podName:fae15b77-656e-4080-a60b-ced44902b861 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:54.862115461 +0000 UTC m=+37.154211802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-f9gqc" (UID: "fae15b77-656e-4080-a60b-ced44902b861") : secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:54.362237 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.362184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9e5cff-51ba-49af-9156-5d6714453eed-tmp-dir\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.362718 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.362691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db6c8ff1-edf0-406d-8438-e502e0a656c7-config\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.362873 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.362849 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/596212fb-ef59-4d1a-9770-86fce25ea36b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.363605 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.363373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-service-ca-bundle\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.363605 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.363572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a719ded6-afb9-4ac3-8c4f-1fea94627815-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:54.364327 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.364275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/596212fb-ef59-4d1a-9770-86fce25ea36b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.364422 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.364336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-serving-cert\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.365105 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.365077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-image-registry-private-configuration\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.365516 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.365496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db6c8ff1-edf0-406d-8438-e502e0a656c7-serving-cert\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.365605 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.365488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-installation-pull-secrets\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.366037 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.366015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205-cert\") pod \"ingress-canary-kmqfs\" (UID: \"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205\") " pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.366141 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.366118 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-stats-auth\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.366417 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.366390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96e39af8-6a4c-42d1-87d0-850180f27c86-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.367482 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.367459 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-default-certificate\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.371564 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.371533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqtfp\" (UniqueName: \"kubernetes.io/projected/fec72562-75b7-41bc-8a75-86fd188e4023-kube-api-access-cqtfp\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:54.371836 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.371559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwqj\" (UniqueName: \"kubernetes.io/projected/3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49-kube-api-access-4zwqj\") pod \"insights-operator-5885fdb759-rhf5t\" (UID: \"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49\") " pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.371836 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.371535 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plh5n\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-kube-api-access-plh5n\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.374307 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.374265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6sh\" (UniqueName: \"kubernetes.io/projected/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-kube-api-access-5z6sh\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.375027 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.374988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9h8\" (UniqueName: \"kubernetes.io/projected/fae15b77-656e-4080-a60b-ced44902b861-kube-api-access-wb9h8\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.381725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.380090 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpfn\" (UniqueName: \"kubernetes.io/projected/fd9e5cff-51ba-49af-9156-5d6714453eed-kube-api-access-rcpfn\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.381725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.380822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2sl\" (UniqueName: \"kubernetes.io/projected/db6c8ff1-edf0-406d-8438-e502e0a656c7-kube-api-access-nf2sl\") pod \"console-operator-584dcb6b54-kqvg8\" (UID: \"db6c8ff1-edf0-406d-8438-e502e0a656c7\") " pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.381725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.381271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-bound-sa-token\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.381725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.381336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk627\" (UniqueName: \"kubernetes.io/projected/596212fb-ef59-4d1a-9770-86fce25ea36b-kube-api-access-vk627\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl\" (UID: \"596212fb-ef59-4d1a-9770-86fce25ea36b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.381725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.381485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbnd\" (UniqueName: \"kubernetes.io/projected/bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205-kube-api-access-rnbnd\") pod \"ingress-canary-kmqfs\" (UID: \"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205\") " pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.381725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.381678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bls5j\" (UniqueName: \"kubernetes.io/projected/96e39af8-6a4c-42d1-87d0-850180f27c86-kube-api-access-bls5j\") pod \"service-ca-operator-5d6c75f84f-rgvpw\" (UID: \"96e39af8-6a4c-42d1-87d0-850180f27c86\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.382112 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.381929 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bq9\" (UniqueName: \"kubernetes.io/projected/34fa7a1c-4bb6-4cef-9b34-105c75ceb894-kube-api-access-t9bq9\") pod \"network-check-source-7d88cb55d8-tfx27\" (UID: \"34fa7a1c-4bb6-4cef-9b34-105c75ceb894\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" Jul 29 03:55:54.382112 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.382058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4cl\" (UniqueName: \"kubernetes.io/projected/525d31bf-6341-4d2d-b238-3313395d1e28-kube-api-access-jk4cl\") pod \"volume-data-source-validator-549c6d54cf-mc8zb\" (UID: \"525d31bf-6341-4d2d-b238-3313395d1e28\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" Jul 29 03:55:54.414480 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.414448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rhf5t" Jul 29 03:55:54.424560 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.424528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" Jul 29 03:55:54.437157 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.437134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" Jul 29 03:55:54.456981 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.456955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" Jul 29 03:55:54.473266 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.473235 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:55:54.503652 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.503622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmqfs" Jul 29 03:55:54.509475 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.509459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" Jul 29 03:55:54.853854 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.853046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb"] Jul 29 03:55:54.865265 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.865226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:54.865414 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.865288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:54.865414 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.865333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.865414 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.865366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:54.865414 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.865403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:54.865650 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.865493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:54.865650 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.865563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:54.865907 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.865737 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:54.865907 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.865809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls podName:fae15b77-656e-4080-a60b-ced44902b861 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:55.865788165 +0000 UTC m=+38.157884507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-f9gqc" (UID: "fae15b77-656e-4080-a60b-ced44902b861") : secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866310 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866389 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:55.86636973 +0000 UTC m=+38.158466087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : secret "router-metrics-certs-default" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866402 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866448 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866440 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert podName:a719ded6-afb9-4ac3-8c4f-1fea94627815 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:55.866427982 +0000 UTC m=+38.158524327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bj6qs" (UID: "a719ded6-afb9-4ac3-8c4f-1fea94627815") : secret "networking-console-plugin-cert" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866482 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls podName:fec72562-75b7-41bc-8a75-86fd188e4023 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:55.866472956 +0000 UTC m=+38.158569295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls") pod "cluster-samples-operator-6748489656-qkfv5" (UID: "fec72562-75b7-41bc-8a75-86fd188e4023") : secret "samples-operator-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866514 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866525 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65679c454d-bn24p: secret "image-registry-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866531 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866555 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls podName:f49df069-1566-41a1-9349-e85601dd1ffa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:55.86654525 +0000 UTC m=+38.158641590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls") pod "image-registry-65679c454d-bn24p" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa") : secret "image-registry-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866573 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls podName:fd9e5cff-51ba-49af-9156-5d6714453eed nodeName:}" failed. No retries permitted until 2026-07-29 03:55:55.866562288 +0000 UTC m=+38.158658629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls") pod "dns-default-n2lxw" (UID: "fd9e5cff-51ba-49af-9156-5d6714453eed") : secret "dns-default-metrics-tls" not found Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.866612 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rhf5t"] Jul 29 03:55:54.867045 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:54.866623 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:55.86660627 +0000 UTC m=+38.158702610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : configmap references non-existent config key: service-ca.crt Jul 29 03:55:54.867761 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:54.867131 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525d31bf_6341_4d2d_b238_3313395d1e28.slice/crio-29df087710672b9faf5b40e87e8a220eb4100ecfff7fd2b68af9ea4e35168a4a WatchSource:0}: Error finding container 29df087710672b9faf5b40e87e8a220eb4100ecfff7fd2b68af9ea4e35168a4a: Status 404 returned error can't find the container with id 29df087710672b9faf5b40e87e8a220eb4100ecfff7fd2b68af9ea4e35168a4a Jul 29 03:55:54.870812 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:54.870790 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9712b1_dd0e_4177_a5c0_4f2a8a5d1e49.slice/crio-3f214fff9b9b29df1a1a17290c24c2cfb031867dff4ea67d4471f450f970c2ed WatchSource:0}: Error finding container 3f214fff9b9b29df1a1a17290c24c2cfb031867dff4ea67d4471f450f970c2ed: Status 404 returned error can't find the container with id 3f214fff9b9b29df1a1a17290c24c2cfb031867dff4ea67d4471f450f970c2ed Jul 29 03:55:54.879270 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.879194 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw"] Jul 29 03:55:54.880444 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.880262 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl"] Jul 29 03:55:54.881536 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.881515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-kqvg8"] Jul 29 03:55:54.882694 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.882671 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmqfs"] Jul 29 03:55:54.885211 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:54.885191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27"] Jul 29 03:55:54.888496 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:54.888473 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbb1392_e8d1_4fdf_9dcb_4a3c689d8205.slice/crio-3b995444d5b55af241560b2f416da86219ff78f9db9a215685c8073ca3243fa3 WatchSource:0}: Error finding container 3b995444d5b55af241560b2f416da86219ff78f9db9a215685c8073ca3243fa3: Status 404 returned error can't find the container with id 3b995444d5b55af241560b2f416da86219ff78f9db9a215685c8073ca3243fa3 Jul 29 03:55:54.889787 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:54.889763 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e39af8_6a4c_42d1_87d0_850180f27c86.slice/crio-9df00b59d768436732de5029317d10d9c7fdcabeacff0c79ac9a30be08d68e5f WatchSource:0}: Error finding container 9df00b59d768436732de5029317d10d9c7fdcabeacff0c79ac9a30be08d68e5f: Status 404 returned error can't find the container with id 9df00b59d768436732de5029317d10d9c7fdcabeacff0c79ac9a30be08d68e5f Jul 29 03:55:54.890777 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:54.890726 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596212fb_ef59_4d1a_9770_86fce25ea36b.slice/crio-5cb03af5ca63a2ead194398fce936ec08ad37705ca415b28eb2cb55eb2644234 WatchSource:0}: Error finding container 5cb03af5ca63a2ead194398fce936ec08ad37705ca415b28eb2cb55eb2644234: Status 404 returned error can't find the container with id 5cb03af5ca63a2ead194398fce936ec08ad37705ca415b28eb2cb55eb2644234 Jul 29 03:55:54.891984 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:54.891825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6c8ff1_edf0_406d_8438_e502e0a656c7.slice/crio-01efd071cb1708e980fc781a3b6e7072b1f57b618befade20956302c1facb01f WatchSource:0}: Error finding container 01efd071cb1708e980fc781a3b6e7072b1f57b618befade20956302c1facb01f: Status 404 returned error can't find the container with id 01efd071cb1708e980fc781a3b6e7072b1f57b618befade20956302c1facb01f Jul 29 03:55:54.892569 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:55:54.892462 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fa7a1c_4bb6_4cef_9b34_105c75ceb894.slice/crio-72c759aab9b1a957e26a7edabb2ec2858806b5dc40ecebec79f01847a4b4db0c WatchSource:0}: Error finding container 72c759aab9b1a957e26a7edabb2ec2858806b5dc40ecebec79f01847a4b4db0c: Status 404 returned error can't find the container with id 72c759aab9b1a957e26a7edabb2ec2858806b5dc40ecebec79f01847a4b4db0c Jul 29 03:55:55.190918 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.190877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:55:55.191116 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.191005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:55:55.191116 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.190901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:55:55.193750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.193724 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 03:55:55.193897 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.193756 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 03:55:55.193897 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.193814 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8jgt\"" Jul 29 03:55:55.193897 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.193758 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-whxkd\"" Jul 29 03:55:55.467322 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.467132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:55.493193 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.493127 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" event={"ID":"96e39af8-6a4c-42d1-87d0-850180f27c86","Type":"ContainerStarted","Data":"9df00b59d768436732de5029317d10d9c7fdcabeacff0c79ac9a30be08d68e5f"} Jul 29 03:55:55.495408 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.495351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" event={"ID":"db6c8ff1-edf0-406d-8438-e502e0a656c7","Type":"ContainerStarted","Data":"01efd071cb1708e980fc781a3b6e7072b1f57b618befade20956302c1facb01f"} Jul 29 03:55:55.500200 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.500076 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b079719-cf88-4c8b-98fe-0f72beec6c97" containerID="a88c579826a0b651d2aa090851027e255be502c0be51d153780f6afe0a680424" exitCode=0 Jul 29 03:55:55.500200 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.500153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerDied","Data":"a88c579826a0b651d2aa090851027e255be502c0be51d153780f6afe0a680424"} Jul 29 03:55:55.502548 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.502474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" event={"ID":"596212fb-ef59-4d1a-9770-86fce25ea36b","Type":"ContainerStarted","Data":"5cb03af5ca63a2ead194398fce936ec08ad37705ca415b28eb2cb55eb2644234"} Jul 29 03:55:55.506823 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.506630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" event={"ID":"525d31bf-6341-4d2d-b238-3313395d1e28","Type":"ContainerStarted","Data":"29df087710672b9faf5b40e87e8a220eb4100ecfff7fd2b68af9ea4e35168a4a"} Jul 29 03:55:55.509058 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.509039 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmqfs" event={"ID":"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205","Type":"ContainerStarted","Data":"3b995444d5b55af241560b2f416da86219ff78f9db9a215685c8073ca3243fa3"} Jul 29 03:55:55.510243 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.510222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" event={"ID":"34fa7a1c-4bb6-4cef-9b34-105c75ceb894","Type":"ContainerStarted","Data":"72c759aab9b1a957e26a7edabb2ec2858806b5dc40ecebec79f01847a4b4db0c"} Jul 29 03:55:55.511341 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.511293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rhf5t" event={"ID":"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49","Type":"ContainerStarted","Data":"3f214fff9b9b29df1a1a17290c24c2cfb031867dff4ea67d4471f450f970c2ed"} Jul 29 03:55:55.529499 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.529344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:55.876210 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.876178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:55.876377 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.876241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:55.876377 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.876280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:55.876488 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.876412 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 03:55:55.876544 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.876441 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 03:55:55.876544 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.876498 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls podName:fd9e5cff-51ba-49af-9156-5d6714453eed nodeName:}" failed. No retries permitted until 2026-07-29 03:55:57.876461813 +0000 UTC m=+40.168558153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls") pod "dns-default-n2lxw" (UID: "fd9e5cff-51ba-49af-9156-5d6714453eed") : secret "dns-default-metrics-tls" not found Jul 29 03:55:55.876650 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.876590 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:57.876549223 +0000 UTC m=+40.168645569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : configmap references non-existent config key: service-ca.crt Jul 29 03:55:55.876650 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.876605 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls podName:fec72562-75b7-41bc-8a75-86fd188e4023 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:57.876597204 +0000 UTC m=+40.168693543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls") pod "cluster-samples-operator-6748489656-qkfv5" (UID: "fec72562-75b7-41bc-8a75-86fd188e4023") : secret "samples-operator-tls" not found Jul 29 03:55:55.876985 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.876963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:55.877077 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.877016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:55.877256 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877238 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 03:55:55.877450 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877423 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 03:55:55.877536 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.877509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:55.877631 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:55.877610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:55.877724 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877713 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:55.877802 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877753 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls podName:fae15b77-656e-4080-a60b-ced44902b861 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:57.877739906 +0000 UTC m=+40.169836252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-f9gqc" (UID: "fae15b77-656e-4080-a60b-ced44902b861") : secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:55.877870 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877812 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 03:55:55.877870 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877823 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65679c454d-bn24p: secret "image-registry-tls" not found Jul 29 03:55:55.877870 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877854 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls podName:f49df069-1566-41a1-9349-e85601dd1ffa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:57.877843026 +0000 UTC m=+40.169939380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls") pod "image-registry-65679c454d-bn24p" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa") : secret "image-registry-tls" not found Jul 29 03:55:55.878033 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877875 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:57.877863621 +0000 UTC m=+40.169959965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : secret "router-metrics-certs-default" not found Jul 29 03:55:55.878033 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:55.877905 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert podName:a719ded6-afb9-4ac3-8c4f-1fea94627815 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:57.877895323 +0000 UTC m=+40.169991679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bj6qs" (UID: "a719ded6-afb9-4ac3-8c4f-1fea94627815") : secret "networking-console-plugin-cert" not found Jul 29 03:55:56.525258 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:56.524316 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b079719-cf88-4c8b-98fe-0f72beec6c97" containerID="d667c7ee9132b516c2e66948d0b03dd732f6d91e0ce9bf4954d0e79ece453278" exitCode=0 Jul 29 03:55:56.525258 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:56.524367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerDied","Data":"d667c7ee9132b516c2e66948d0b03dd732f6d91e0ce9bf4954d0e79ece453278"} Jul 29 03:55:57.533649 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.533154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" event={"ID":"4b079719-cf88-4c8b-98fe-0f72beec6c97","Type":"ContainerStarted","Data":"dc1ee16b551666455eb8de835cd5ba54a1b4355dfe70ea290a380a3f4a74835d"} Jul 29 03:55:57.535502 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.535474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" event={"ID":"525d31bf-6341-4d2d-b238-3313395d1e28","Type":"ContainerStarted","Data":"07de79306636df9a0d756b82c6a18116f17f676424b04aa0e6fbe11ec6db6a9e"} Jul 29 03:55:57.556918 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.556667 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9gt4v" podStartSLOduration=10.565131461 podStartE2EDuration="39.556646637s" totalImagesPullingTime="28.991515176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:57.554020686 +0000 UTC m=+39.846117049" watchObservedRunningTime="2026-07-29 03:55:57.556646637 +0000 UTC m=+39.848743002" Jul 29 03:55:57.682415 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.682334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:57.737103 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.737064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:57.898645 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.898600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:55:57.898816 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.898670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:55:57.898816 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.898768 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:57.898816 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.898779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:55:57.899012 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.898828 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 03:55:57.899012 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.898846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:57.899012 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.898836 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls podName:fae15b77-656e-4080-a60b-ced44902b861 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:01.898820469 +0000 UTC m=+44.190916809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-f9gqc" (UID: "fae15b77-656e-4080-a60b-ced44902b861") : secret "cluster-monitoring-operator-tls" not found Jul 29 03:55:57.899012 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.898931 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls podName:fec72562-75b7-41bc-8a75-86fd188e4023 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:01.89891249 +0000 UTC m=+44.191008831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls") pod "cluster-samples-operator-6748489656-qkfv5" (UID: "fec72562-75b7-41bc-8a75-86fd188e4023") : secret "samples-operator-tls" not found Jul 29 03:55:57.899012 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.898956 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 03:55:57.899012 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.898966 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:01.898949579 +0000 UTC m=+44.191045919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : configmap references non-existent config key: service-ca.crt Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.899021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.899053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899066 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls podName:fd9e5cff-51ba-49af-9156-5d6714453eed nodeName:}" failed. No retries permitted until 2026-07-29 03:56:01.899053279 +0000 UTC m=+44.191149634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls") pod "dns-default-n2lxw" (UID: "fd9e5cff-51ba-49af-9156-5d6714453eed") : secret "dns-default-metrics-tls" not found Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899088 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899107 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899133 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:01.899122642 +0000 UTC m=+44.191218981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : secret "router-metrics-certs-default" not found Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:57.899130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899150 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert podName:a719ded6-afb9-4ac3-8c4f-1fea94627815 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:01.899141443 +0000 UTC m=+44.191237786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bj6qs" (UID: "a719ded6-afb9-4ac3-8c4f-1fea94627815") : secret "networking-console-plugin-cert" not found Jul 29 03:55:57.899253 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899252 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 03:55:57.899602 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899266 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65679c454d-bn24p: secret "image-registry-tls" not found Jul 29 03:55:57.899602 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:55:57.899328 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls podName:f49df069-1566-41a1-9349-e85601dd1ffa nodeName:}" failed. No retries permitted until 2026-07-29 03:56:01.899315529 +0000 UTC m=+44.191411884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls") pod "image-registry-65679c454d-bn24p" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa") : secret "image-registry-tls" not found Jul 29 03:55:58.247315 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:58.247230 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:58.302510 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:58.302467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:58.936674 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:58.936636 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:58.990045 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:58.990009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:59.274094 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:59.274020 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:59.373143 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:59.373106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:59.813420 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:59.813386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:55:59.868335 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:55:59.868295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:00.222863 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:00.222777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:00.274974 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:00.274935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:00.649769 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:00.649738 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:00.701356 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:00.701320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:01.230798 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.230755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:01.936427 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.936392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:56:01.936636 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.936466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:56:01.936636 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.936513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:56:01.936636 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.936554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:56:01.936636 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936579 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 03:56:01.936636 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936605 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65679c454d-bn24p: secret "image-registry-tls" not found Jul 29 03:56:01.936636 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936628 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936648 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.936587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936672 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls podName:f49df069-1566-41a1-9349-e85601dd1ffa nodeName:}" failed. No retries permitted until 2026-07-29 03:56:09.936649724 +0000 UTC m=+52.228746088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls") pod "image-registry-65679c454d-bn24p" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa") : secret "image-registry-tls" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936580 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936707 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls podName:fec72562-75b7-41bc-8a75-86fd188e4023 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:09.936690068 +0000 UTC m=+52.228786409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls") pod "cluster-samples-operator-6748489656-qkfv5" (UID: "fec72562-75b7-41bc-8a75-86fd188e4023") : secret "samples-operator-tls" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936741 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls podName:fd9e5cff-51ba-49af-9156-5d6714453eed nodeName:}" failed. No retries permitted until 2026-07-29 03:56:09.936731351 +0000 UTC m=+52.228827692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls") pod "dns-default-n2lxw" (UID: "fd9e5cff-51ba-49af-9156-5d6714453eed") : secret "dns-default-metrics-tls" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936759 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls podName:fae15b77-656e-4080-a60b-ced44902b861 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:09.936750727 +0000 UTC m=+52.228847069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-f9gqc" (UID: "fae15b77-656e-4080-a60b-ced44902b861") : secret "cluster-monitoring-operator-tls" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936776 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:09.936768632 +0000 UTC m=+52.228864972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : configmap references non-existent config key: service-ca.crt Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.936806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:01.936841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936953 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 03:56:01.936953 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936954 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 03:56:01.937437 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.936996 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert podName:a719ded6-afb9-4ac3-8c4f-1fea94627815 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:09.936984745 +0000 UTC m=+52.229081086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bj6qs" (UID: "a719ded6-afb9-4ac3-8c4f-1fea94627815") : secret "networking-console-plugin-cert" not found Jul 29 03:56:01.937437 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:01.937011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:09.937002952 +0000 UTC m=+52.229099300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : secret "router-metrics-certs-default" not found Jul 29 03:56:03.555102 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.555031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rhf5t" event={"ID":"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49","Type":"ContainerStarted","Data":"6e07f3190d0b28216f2211a869cf74210b5eaac6c9e9f9ffd6b1518f6de688ce"} Jul 29 03:56:03.556786 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.556758 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" event={"ID":"96e39af8-6a4c-42d1-87d0-850180f27c86","Type":"ContainerStarted","Data":"75fc396a29f8f3073194aab8a7982829544d65d331fe58997396ae7a023384af"} Jul 29 03:56:03.558624 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.558607 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/0.log" Jul 29 03:56:03.558740 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.558645 2576 generic.go:356] "Generic (PLEG): container finished" podID="db6c8ff1-edf0-406d-8438-e502e0a656c7" containerID="c75069a9fe2c126db77c88a5d743a6d6006b83ed1186cc32a80b68989d47706e" exitCode=255 Jul 29 03:56:03.558740 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.558726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" event={"ID":"db6c8ff1-edf0-406d-8438-e502e0a656c7","Type":"ContainerDied","Data":"c75069a9fe2c126db77c88a5d743a6d6006b83ed1186cc32a80b68989d47706e"} Jul 29 03:56:03.558916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.558872 2576 scope.go:117] "RemoveContainer" containerID="c75069a9fe2c126db77c88a5d743a6d6006b83ed1186cc32a80b68989d47706e" Jul 29 03:56:03.560492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.560460 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" event={"ID":"596212fb-ef59-4d1a-9770-86fce25ea36b","Type":"ContainerStarted","Data":"9a47659f8e0bd25d897c5056e459b8278216f302ff8274284c8379bb93d0ead9"} Jul 29 03:56:03.562378 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.562350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmqfs" event={"ID":"bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205","Type":"ContainerStarted","Data":"01db91aed925c146e9082257ab820dd360b7d6a200e75e80879ae9c3748d9220"} Jul 29 03:56:03.563914 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.563870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" event={"ID":"34fa7a1c-4bb6-4cef-9b34-105c75ceb894","Type":"ContainerStarted","Data":"aa82a40c8371259a2f7315eee18e1e6bb8ad0adcf79ffbb0c0c356020410df1f"} Jul 29 03:56:03.575171 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.574779 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mc8zb" podStartSLOduration=19.565120882 podStartE2EDuration="21.574761177s" totalImagesPullingTime="2.009640295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:57.570056993 +0000 UTC m=+39.862153356" watchObservedRunningTime="2026-07-29 03:56:03.574761177 +0000 UTC m=+45.866857520" Jul 29 03:56:03.575295 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.575158 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-rhf5t" podStartSLOduration=13.771478538 podStartE2EDuration="21.575149077s" totalImagesPullingTime="7.803670539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:03.574052156 +0000 UTC m=+45.866148513" watchObservedRunningTime="2026-07-29 03:56:03.575149077 +0000 UTC m=+45.867245420" Jul 29 03:56:03.595087 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.593938 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" podStartSLOduration=12.809621581 podStartE2EDuration="20.593916794s" totalImagesPullingTime="7.784295213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:03.592784182 +0000 UTC m=+45.884880542" watchObservedRunningTime="2026-07-29 03:56:03.593916794 +0000 UTC m=+45.886013159" Jul 29 03:56:03.614354 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.614303 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" podStartSLOduration=12.830781706 podStartE2EDuration="20.614280334s" totalImagesPullingTime="7.783498628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:03.612652383 +0000 UTC m=+45.904748743" watchObservedRunningTime="2026-07-29 03:56:03.614280334 +0000 UTC m=+45.906376697" Jul 29 03:56:03.648278 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.648210 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-tfx27" podStartSLOduration=12.855518392 podStartE2EDuration="20.648190475s" totalImagesPullingTime="7.792672083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:03.647130412 +0000 UTC m=+45.939226775" watchObservedRunningTime="2026-07-29 03:56:03.648190475 +0000 UTC m=+45.940286836" Jul 29 03:56:03.665067 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.665002 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kmqfs" podStartSLOduration=1.878935072 podStartE2EDuration="9.66498117s" totalImagesPullingTime="7.786046098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:03.664802325 +0000 UTC m=+45.956898689" watchObservedRunningTime="2026-07-29 03:56:03.66498117 +0000 UTC m=+45.957077531" Jul 29 03:56:03.947165 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:03.947130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:04.011222 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.011188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:04.265639 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.265552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:04.327509 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.327474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:04.402162 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.402133 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9c4g7"] Jul 29 03:56:04.418102 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.418069 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9c4g7"] Jul 29 03:56:04.418259 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.418195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.420821 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.420796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 03:56:04.420821 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.420796 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 03:56:04.421015 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.420978 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kz6jn\"" Jul 29 03:56:04.428197 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.428171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:04.463915 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.463854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-crio-socket\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.464069 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.464029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-data-volume\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.464069 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.464057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.464190 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.464170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.464287 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.464269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xws5h\" (UniqueName: \"kubernetes.io/projected/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-api-access-xws5h\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.473736 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.473715 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:56:04.473827 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.473759 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:56:04.511603 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.511577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:04.564745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.564682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xws5h\" (UniqueName: \"kubernetes.io/projected/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-api-access-xws5h\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.564745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.564741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-crio-socket\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.565196 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.564815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-data-volume\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.565196 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.565022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-crio-socket\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.565196 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.565049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.565196 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:04.565188 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 03:56:04.565384 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.565211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.565384 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.565222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-data-volume\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.565384 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:04.565248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:05.065228493 +0000 UTC m=+47.357324859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : configmap "kube-rbac-proxy" not found Jul 29 03:56:04.565384 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:04.565300 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 03:56:04.565384 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:04.565347 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:05.065332418 +0000 UTC m=+47.357428765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : secret "insights-runtime-extractor-tls" not found Jul 29 03:56:04.569214 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.569196 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 03:56:04.569604 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.569589 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/0.log" Jul 29 03:56:04.569672 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.569628 2576 generic.go:356] "Generic (PLEG): container finished" podID="db6c8ff1-edf0-406d-8438-e502e0a656c7" containerID="b8c3e6e7694e21fae0bd4f108fd7bdc04f7d8520b571ba6f82410750ce05b233" exitCode=255 Jul 29 03:56:04.569717 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.569696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" event={"ID":"db6c8ff1-edf0-406d-8438-e502e0a656c7","Type":"ContainerDied","Data":"b8c3e6e7694e21fae0bd4f108fd7bdc04f7d8520b571ba6f82410750ce05b233"} Jul 29 03:56:04.569770 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.569756 2576 scope.go:117] "RemoveContainer" containerID="c75069a9fe2c126db77c88a5d743a6d6006b83ed1186cc32a80b68989d47706e" Jul 29 03:56:04.570533 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.569998 2576 scope.go:117] "RemoveContainer" containerID="b8c3e6e7694e21fae0bd4f108fd7bdc04f7d8520b571ba6f82410750ce05b233" Jul 29 03:56:04.570533 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:04.570220 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-kqvg8_openshift-console-operator(db6c8ff1-edf0-406d-8438-e502e0a656c7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" podUID="db6c8ff1-edf0-406d-8438-e502e0a656c7" Jul 29 03:56:04.577271 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.577252 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xws5h\" (UniqueName: \"kubernetes.io/projected/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-api-access-xws5h\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:04.590382 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.590356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:04.973240 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:04.973203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:05.031599 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:05.031566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:05.069984 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:05.069959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:05.070080 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:05.070016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:05.070080 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:05.070069 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 03:56:05.070150 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:05.070115 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 03:56:05.070150 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:05.070129 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:06.070114481 +0000 UTC m=+48.362210841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : configmap "kube-rbac-proxy" not found Jul 29 03:56:05.070218 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:05.070163 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:06.070151083 +0000 UTC m=+48.362247422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : secret "insights-runtime-extractor-tls" not found Jul 29 03:56:05.575397 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:05.575372 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 03:56:05.575746 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:05.575688 2576 scope.go:117] "RemoveContainer" containerID="b8c3e6e7694e21fae0bd4f108fd7bdc04f7d8520b571ba6f82410750ce05b233" Jul 29 03:56:05.575900 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:05.575858 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-kqvg8_openshift-console-operator(db6c8ff1-edf0-406d-8438-e502e0a656c7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" podUID="db6c8ff1-edf0-406d-8438-e502e0a656c7" Jul 29 03:56:06.078812 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:06.078782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:06.078984 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:06.078844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:06.078984 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:06.078911 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 03:56:06.079057 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:06.078987 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:08.078971835 +0000 UTC m=+50.371068174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : configmap "kube-rbac-proxy" not found Jul 29 03:56:06.079057 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:06.078991 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 03:56:06.079057 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:06.079037 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:08.079025366 +0000 UTC m=+50.371121706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : secret "insights-runtime-extractor-tls" not found Jul 29 03:56:06.685984 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:06.685951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:06.787586 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:06.787553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:06.986849 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:06.986788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd8bl_0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b/dns-node-resolver/0.log" Jul 29 03:56:07.491563 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:07.491531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:07.987094 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:07.987069 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnhv8_e1d96163-4e1b-4d7f-ad07-55d7488acb5e/node-ca/0.log" Jul 29 03:56:08.095182 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:08.094953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:08.095182 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:08.095056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:08.095373 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:08.095254 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 03:56:08.095373 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:08.095324 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:12.095297258 +0000 UTC m=+54.387393601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : secret "insights-runtime-extractor-tls" not found Jul 29 03:56:08.097991 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:08.095742 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 03:56:08.097991 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:08.095805 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:12.095787867 +0000 UTC m=+54.387884221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : configmap "kube-rbac-proxy" not found Jul 29 03:56:08.386759 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:08.386729 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmqfs_bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205/serve-healthcheck-canary/0.log" Jul 29 03:56:09.189280 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:09.189246 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl_596212fb-ef59-4d1a-9770-86fce25ea36b/kube-storage-version-migrator-operator/0.log" Jul 29 03:56:10.012376 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.012339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:10.012554 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.012379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:10.012554 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.012424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:56:10.012554 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.012454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:56:10.012554 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.012505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012560 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:26.012542461 +0000 UTC m=+68.304638801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : configmap references non-existent config key: service-ca.crt Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012585 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.012591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012586 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012652 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert podName:a719ded6-afb9-4ac3-8c4f-1fea94627815 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:26.012640142 +0000 UTC m=+68.304736482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bj6qs" (UID: "a719ded6-afb9-4ac3-8c4f-1fea94627815") : secret "networking-console-plugin-cert" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.012635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012601 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012689 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012699 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs podName:28ed722e-2b3c-4d7e-90ed-5f247f1b0f46 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:26.012682451 +0000 UTC m=+68.304778791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs") pod "router-default-77f89b64c-jk642" (UID: "28ed722e-2b3c-4d7e-90ed-5f247f1b0f46") : secret "router-metrics-certs-default" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012619 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012716 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65679c454d-bn24p: secret "image-registry-tls" not found Jul 29 03:56:10.012745 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012724 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 03:56:10.013201 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012730 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls podName:fae15b77-656e-4080-a60b-ced44902b861 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:26.012721543 +0000 UTC m=+68.304817882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-f9gqc" (UID: "fae15b77-656e-4080-a60b-ced44902b861") : secret "cluster-monitoring-operator-tls" not found Jul 29 03:56:10.013201 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012824 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls podName:fd9e5cff-51ba-49af-9156-5d6714453eed nodeName:}" failed. No retries permitted until 2026-07-29 03:56:26.012797547 +0000 UTC m=+68.304893895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls") pod "dns-default-n2lxw" (UID: "fd9e5cff-51ba-49af-9156-5d6714453eed") : secret "dns-default-metrics-tls" not found Jul 29 03:56:10.013201 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012855 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls podName:f49df069-1566-41a1-9349-e85601dd1ffa nodeName:}" failed. No retries permitted until 2026-07-29 03:56:26.012844247 +0000 UTC m=+68.304940587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls") pod "image-registry-65679c454d-bn24p" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa") : secret "image-registry-tls" not found Jul 29 03:56:10.013201 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:10.012898 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls podName:fec72562-75b7-41bc-8a75-86fd188e4023 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:26.012869494 +0000 UTC m=+68.304965837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls") pod "cluster-samples-operator-6748489656-qkfv5" (UID: "fec72562-75b7-41bc-8a75-86fd188e4023") : secret "samples-operator-tls" not found Jul 29 03:56:10.142709 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.142670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:10.576925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.576870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:10.733636 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.733605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:10.835277 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.835191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:10.939111 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:10.939084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:11.039740 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:11.039713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:11.481146 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:11.481113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:12.071836 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.071797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:12.131854 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.131826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:12.132003 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.131910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:12.132056 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:12.132037 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 03:56:12.132101 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:12.132093 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls podName:12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:20.132080361 +0000 UTC m=+62.424176701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9c4g7" (UID: "12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7") : secret "insights-runtime-extractor-tls" not found Jul 29 03:56:12.132393 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.132376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:12.471061 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.471031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:12.698400 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.698370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:12.756036 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.755958 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:12.874469 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:12.874430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:13.071384 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:13.071297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:13.469283 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:13.469256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.004373 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.004340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.103559 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.103528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.152807 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.152778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.266706 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.266634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.462698 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.462665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.473501 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.473469 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:56:14.473793 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.473529 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:56:14.473844 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.473832 2576 scope.go:117] "RemoveContainer" containerID="b8c3e6e7694e21fae0bd4f108fd7bdc04f7d8520b571ba6f82410750ce05b233" Jul 29 03:56:14.563905 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.563844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.605318 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.605295 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 03:56:14.605438 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.605342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" event={"ID":"db6c8ff1-edf0-406d-8438-e502e0a656c7","Type":"ContainerStarted","Data":"7e7dc5a08217bf3635e6a169cbeeaa4f6cc38d19933f2b89326a03ec2414fec9"} Jul 29 03:56:14.605721 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.605701 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:56:14.621784 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.621736 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" podStartSLOduration=23.839047005 podStartE2EDuration="31.621722295s" totalImagesPullingTime="7.78267529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:14.621352515 +0000 UTC m=+56.913448877" watchObservedRunningTime="2026-07-29 03:56:14.621722295 +0000 UTC m=+56.913818726" Jul 29 03:56:14.672536 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.672511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:14.952476 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:14.952455 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-kqvg8" Jul 29 03:56:15.143821 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:15.143785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:15.257487 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:15.257407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:15.347910 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:15.347847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:15.447633 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:15.447601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:15.669793 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:15.669760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:15.806288 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:15.806258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:15.906259 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:15.906225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.082321 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.082244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.179336 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.179305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.225757 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.225729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.471668 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.471633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.601761 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.601729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.706540 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.706497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.803520 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.803447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:16.903707 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:16.903676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:17.004650 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:17.004625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:17.105137 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:17.105072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:17.164397 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:17.164372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:17.217845 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:17.217813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:17.474958 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:17.474917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:17.626533 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:17.626503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:17.808282 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:17.808200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:18.274346 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:18.274315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:18.468669 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:18.468630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:18.870194 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:18.870151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:19.079247 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:19.079220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:19.180485 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:19.180457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:19.484144 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:19.484079 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxm72" Jul 29 03:56:19.671663 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:19.671628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:20.197973 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.197938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:20.200645 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.200624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9c4g7\" (UID: \"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7\") " pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:20.272217 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.272188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:20.330979 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.330955 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kz6jn\"" Jul 29 03:56:20.339456 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.339428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9c4g7" Jul 29 03:56:20.465372 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.465306 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9c4g7"] Jul 29 03:56:20.468177 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:56:20.468152 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b6a5a8_bb22_47f5_8018_0e15f4ca4bb7.slice/crio-85fca49734e298be84a3037360a05f73985aaf2f3982082350cccf53e9c32dec WatchSource:0}: Error finding container 85fca49734e298be84a3037360a05f73985aaf2f3982082350cccf53e9c32dec: Status 404 returned error can't find the container with id 85fca49734e298be84a3037360a05f73985aaf2f3982082350cccf53e9c32dec Jul 29 03:56:20.474023 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.473998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:20.624984 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.624948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4g7" event={"ID":"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7","Type":"ContainerStarted","Data":"85fca49734e298be84a3037360a05f73985aaf2f3982082350cccf53e9c32dec"} Jul 29 03:56:20.665613 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:20.665585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:21.075408 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:21.075373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:21.479157 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:21.479102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:21.678349 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:21.678306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:22.631684 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:22.631639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:22.688196 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:22.688166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:23.932954 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:23.932917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:56:23.935899 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:23.935860 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 03:56:23.945824 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:23.945805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e976f8e0-d011-47d3-a2c4-61198b51f2b9-metrics-certs\") pod \"network-metrics-daemon-qr5hw\" (UID: \"e976f8e0-d011-47d3-a2c4-61198b51f2b9\") " pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:56:24.002690 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.002664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-whxkd\"" Jul 29 03:56:24.010906 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.010873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qr5hw" Jul 29 03:56:24.034259 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.034228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:56:24.036637 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.036620 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 03:56:24.046939 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.046912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/525a2c3e-cca1-435f-a3d6-7b94f465d5f5-original-pull-secret\") pod \"global-pull-secret-syncer-x6mqc\" (UID: \"525a2c3e-cca1-435f-a3d6-7b94f465d5f5\") " pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:56:24.134976 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.134940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:56:24.137966 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.137940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzwf\" (UniqueName: \"kubernetes.io/projected/4d3974fa-a38b-499b-b42a-c9abc7b6f6af-kube-api-access-pzzwf\") pod \"network-check-target-xznfq\" (UID: \"4d3974fa-a38b-499b-b42a-c9abc7b6f6af\") " pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:56:24.309030 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.308959 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8jgt\"" Jul 29 03:56:24.311045 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.311017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6mqc" Jul 29 03:56:24.316629 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:24.316606 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:56:25.499270 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:25.499195 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qr5hw"] Jul 29 03:56:25.501654 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:56:25.501628 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode976f8e0_d011_47d3_a2c4_61198b51f2b9.slice/crio-ed48b6f270efb004a9ac31dc20039283cf3f3f1e778334bdaa1ff016661b1a86 WatchSource:0}: Error finding container ed48b6f270efb004a9ac31dc20039283cf3f3f1e778334bdaa1ff016661b1a86: Status 404 returned error can't find the container with id ed48b6f270efb004a9ac31dc20039283cf3f3f1e778334bdaa1ff016661b1a86 Jul 29 03:56:25.643465 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:25.643433 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qr5hw" event={"ID":"e976f8e0-d011-47d3-a2c4-61198b51f2b9","Type":"ContainerStarted","Data":"ed48b6f270efb004a9ac31dc20039283cf3f3f1e778334bdaa1ff016661b1a86"} Jul 29 03:56:25.715319 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:25.715292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6mqc"] Jul 29 03:56:25.717846 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:25.717826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xznfq"] Jul 29 03:56:25.718570 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:56:25.718551 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525a2c3e_cca1_435f_a3d6_7b94f465d5f5.slice/crio-64c0d27dd2a6d076cc82efd38ebfbb6c65f1bf2a882c697803d5c4b1df2911e0 WatchSource:0}: Error finding container 64c0d27dd2a6d076cc82efd38ebfbb6c65f1bf2a882c697803d5c4b1df2911e0: Status 404 returned error can't find the container with id 64c0d27dd2a6d076cc82efd38ebfbb6c65f1bf2a882c697803d5c4b1df2911e0 Jul 29 03:56:25.721261 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:56:25.721236 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3974fa_a38b_499b_b42a_c9abc7b6f6af.slice/crio-3d895927f79ee49f979cf3c257f027af22f45eef46619fca091e09e1d3aebaad WatchSource:0}: Error finding container 3d895927f79ee49f979cf3c257f027af22f45eef46619fca091e09e1d3aebaad: Status 404 returned error can't find the container with id 3d895927f79ee49f979cf3c257f027af22f45eef46619fca091e09e1d3aebaad Jul 29 03:56:26.054519 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.054435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:56:26.054519 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.054507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:56:26.054758 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.054549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:56:26.054758 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.054591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:26.054758 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.054623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:26.054758 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.054657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:56:26.054758 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.054714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:56:26.055530 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.055495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-service-ca-bundle\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:26.057674 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.057645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"image-registry-65679c454d-bn24p\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:56:26.057784 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.057653 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd9e5cff-51ba-49af-9156-5d6714453eed-metrics-tls\") pod \"dns-default-n2lxw\" (UID: \"fd9e5cff-51ba-49af-9156-5d6714453eed\") " pod="openshift-dns/dns-default-n2lxw" Jul 29 03:56:26.057830 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.057784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fae15b77-656e-4080-a60b-ced44902b861-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-f9gqc\" (UID: \"fae15b77-656e-4080-a60b-ced44902b861\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:56:26.058165 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.058145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ed722e-2b3c-4d7e-90ed-5f247f1b0f46-metrics-certs\") pod \"router-default-77f89b64c-jk642\" (UID: \"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46\") " pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:26.058532 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.058513 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719ded6-afb9-4ac3-8c4f-1fea94627815-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bj6qs\" (UID: \"a719ded6-afb9-4ac3-8c4f-1fea94627815\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:56:26.059594 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.059572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec72562-75b7-41bc-8a75-86fd188e4023-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qkfv5\" (UID: \"fec72562-75b7-41bc-8a75-86fd188e4023\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:56:26.234413 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.234370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:26.250742 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.250716 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wjq2b\"" Jul 29 03:56:26.258218 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.258178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:56:26.268804 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.268783 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hj7p6\"" Jul 29 03:56:26.276785 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.276759 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" Jul 29 03:56:26.282127 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.282108 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhgfp\"" Jul 29 03:56:26.285140 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.285109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:26.289953 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.289936 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gdxwb\"" Jul 29 03:56:26.290117 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.290081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2lxw" Jul 29 03:56:26.297942 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.297924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" Jul 29 03:56:26.298184 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.298162 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8tqn7\"" Jul 29 03:56:26.306301 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.306183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:26.318223 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.318201 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nxb4k\"" Jul 29 03:56:26.325706 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.325658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" Jul 29 03:56:26.651297 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.651214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4g7" event={"ID":"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7","Type":"ContainerStarted","Data":"f0cc578a30271ccff4cafcdf3014cc79ddac2ea7758844749ed8a2119f4c4d1b"} Jul 29 03:56:26.654079 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.653770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xznfq" event={"ID":"4d3974fa-a38b-499b-b42a-c9abc7b6f6af","Type":"ContainerStarted","Data":"436f1c44529d8c508c91c8688078297c66cec89b044698784536380f53f4275b"} Jul 29 03:56:26.654079 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.653837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xznfq" event={"ID":"4d3974fa-a38b-499b-b42a-c9abc7b6f6af","Type":"ContainerStarted","Data":"3d895927f79ee49f979cf3c257f027af22f45eef46619fca091e09e1d3aebaad"} Jul 29 03:56:26.654079 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.654039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:56:26.657298 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.657269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6mqc" event={"ID":"525a2c3e-cca1-435f-a3d6-7b94f465d5f5","Type":"ContainerStarted","Data":"64c0d27dd2a6d076cc82efd38ebfbb6c65f1bf2a882c697803d5c4b1df2911e0"} Jul 29 03:56:26.844980 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.841366 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xznfq" podStartSLOduration=68.841342178 podStartE2EDuration="1m8.841342178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:26.685161389 +0000 UTC m=+68.977257832" watchObservedRunningTime="2026-07-29 03:56:26.841342178 +0000 UTC m=+69.133438543" Jul 29 03:56:26.844980 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.841825 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p"] Jul 29 03:56:26.862677 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.860387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.863218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1526c58-9b87-4c23-b006-f84bed8f99a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p\" (UID: \"b1526c58-9b87-4c23-b006-f84bed8f99a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.863296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgs7\" (UniqueName: \"kubernetes.io/projected/b1526c58-9b87-4c23-b006-f84bed8f99a1-kube-api-access-kpgs7\") pod \"managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p\" (UID: \"b1526c58-9b87-4c23-b006-f84bed8f99a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.863339 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5s6nc\"" Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.864121 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.864358 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.864578 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.866433 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p"] Jul 29 03:56:26.872647 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.868667 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 29 03:56:26.926023 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.925993 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-n2dwp"] Jul 29 03:56:26.962409 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.962383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65679c454d-bn24p"] Jul 29 03:56:26.962616 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.962561 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-n2dwp"] Jul 29 03:56:26.962834 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.962818 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-n2dwp" Jul 29 03:56:26.970331 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.969860 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ncrzt\"" Jul 29 03:56:26.970331 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.970162 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 03:56:26.970694 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.970518 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 03:56:26.973873 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.973213 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1526c58-9b87-4c23-b006-f84bed8f99a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p\" (UID: \"b1526c58-9b87-4c23-b006-f84bed8f99a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:26.973873 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.973215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1526c58-9b87-4c23-b006-f84bed8f99a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p\" (UID: \"b1526c58-9b87-4c23-b006-f84bed8f99a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:26.973873 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.973352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgs7\" (UniqueName: \"kubernetes.io/projected/b1526c58-9b87-4c23-b006-f84bed8f99a1-kube-api-access-kpgs7\") pod \"managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p\" (UID: \"b1526c58-9b87-4c23-b006-f84bed8f99a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:26.976866 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.976845 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5"] Jul 29 03:56:26.979061 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.978825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:26.990693 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:26.990624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgs7\" (UniqueName: \"kubernetes.io/projected/b1526c58-9b87-4c23-b006-f84bed8f99a1-kube-api-access-kpgs7\") pod \"managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p\" (UID: \"b1526c58-9b87-4c23-b006-f84bed8f99a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:27.002492 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.002423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2lxw"] Jul 29 03:56:27.008781 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.008760 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs"] Jul 29 03:56:27.042575 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.042539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77f89b64c-jk642"] Jul 29 03:56:27.064936 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:56:27.064905 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49df069_1566_41a1_9349_e85601dd1ffa.slice/crio-cd23d78fb6db2b71b7d6015fd6243c77c18e7d46f609903f25d975048f2ecd50 WatchSource:0}: Error finding container cd23d78fb6db2b71b7d6015fd6243c77c18e7d46f609903f25d975048f2ecd50: Status 404 returned error can't find the container with id cd23d78fb6db2b71b7d6015fd6243c77c18e7d46f609903f25d975048f2ecd50 Jul 29 03:56:27.069461 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.069311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:27.075339 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.075137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkn6\" (UniqueName: \"kubernetes.io/projected/9a404653-f311-4db0-8c78-10e4e9d73000-kube-api-access-vrkn6\") pod \"downloads-7c97dc4555-n2dwp\" (UID: \"9a404653-f311-4db0-8c78-10e4e9d73000\") " pod="openshift-console/downloads-7c97dc4555-n2dwp" Jul 29 03:56:27.163868 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.163768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:27.177040 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.176713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkn6\" (UniqueName: \"kubernetes.io/projected/9a404653-f311-4db0-8c78-10e4e9d73000-kube-api-access-vrkn6\") pod \"downloads-7c97dc4555-n2dwp\" (UID: \"9a404653-f311-4db0-8c78-10e4e9d73000\") " pod="openshift-console/downloads-7c97dc4555-n2dwp" Jul 29 03:56:27.191965 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.191875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkn6\" (UniqueName: \"kubernetes.io/projected/9a404653-f311-4db0-8c78-10e4e9d73000-kube-api-access-vrkn6\") pod \"downloads-7c97dc4555-n2dwp\" (UID: \"9a404653-f311-4db0-8c78-10e4e9d73000\") " pod="openshift-console/downloads-7c97dc4555-n2dwp" Jul 29 03:56:27.227806 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.227245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" Jul 29 03:56:27.256485 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.254821 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc"] Jul 29 03:56:27.259305 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.259117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:27.287805 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.287711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-n2dwp" Jul 29 03:56:27.434989 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.432996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p"] Jul 29 03:56:27.502355 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.502332 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-n2dwp"] Jul 29 03:56:27.505251 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:56:27.505216 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a404653_f311_4db0_8c78_10e4e9d73000.slice/crio-39dc10f70edc27a6971293026e06d1b8d8560fbbaaedcecc1a0ecdb7d26e7dfc WatchSource:0}: Error finding container 39dc10f70edc27a6971293026e06d1b8d8560fbbaaedcecc1a0ecdb7d26e7dfc: Status 404 returned error can't find the container with id 39dc10f70edc27a6971293026e06d1b8d8560fbbaaedcecc1a0ecdb7d26e7dfc Jul 29 03:56:27.670435 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.670368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2lxw" event={"ID":"fd9e5cff-51ba-49af-9156-5d6714453eed","Type":"ContainerStarted","Data":"ae3b6156e21c1bc88cd5b77d18fa8b193c0ca3b4ac6dec16bf4e384ba554f655"} Jul 29 03:56:27.672824 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.672797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65679c454d-bn24p" event={"ID":"f49df069-1566-41a1-9349-e85601dd1ffa","Type":"ContainerStarted","Data":"e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5"} Jul 29 03:56:27.672984 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.672857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65679c454d-bn24p" event={"ID":"f49df069-1566-41a1-9349-e85601dd1ffa","Type":"ContainerStarted","Data":"cd23d78fb6db2b71b7d6015fd6243c77c18e7d46f609903f25d975048f2ecd50"} Jul 29 03:56:27.673150 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.673102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:56:27.674934 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.674786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-n2dwp" event={"ID":"9a404653-f311-4db0-8c78-10e4e9d73000","Type":"ContainerStarted","Data":"39dc10f70edc27a6971293026e06d1b8d8560fbbaaedcecc1a0ecdb7d26e7dfc"} Jul 29 03:56:27.676991 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.676912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" event={"ID":"fec72562-75b7-41bc-8a75-86fd188e4023","Type":"ContainerStarted","Data":"e45b81e1dc355dd256ffdfeca14caac4a949539807e01f6cc444ef7562798da8"} Jul 29 03:56:27.678923 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.678397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" event={"ID":"b1526c58-9b87-4c23-b006-f84bed8f99a1","Type":"ContainerStarted","Data":"d6bd41fda9ab4139db4e7e96c365db0ad3ba22274d9b8c6f38e97a87fa94d144"} Jul 29 03:56:27.680873 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.680845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" event={"ID":"fae15b77-656e-4080-a60b-ced44902b861","Type":"ContainerStarted","Data":"b08fe812e922aba70a5550a274da0122adf7c08bc5a51ebfb7b4caab725e2554"} Jul 29 03:56:27.684697 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.684676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qr5hw" event={"ID":"e976f8e0-d011-47d3-a2c4-61198b51f2b9","Type":"ContainerStarted","Data":"69233e23550a278aa79acc8e9a566674113494e1c9964ef4373d39565d69271c"} Jul 29 03:56:27.684818 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.684718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qr5hw" event={"ID":"e976f8e0-d011-47d3-a2c4-61198b51f2b9","Type":"ContainerStarted","Data":"9c873427c1c2bedb079c1d61eee42dc324ce2938cacc9f18b9aaf7c0800f01b6"} Jul 29 03:56:27.686706 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.686682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" event={"ID":"a719ded6-afb9-4ac3-8c4f-1fea94627815","Type":"ContainerStarted","Data":"1542e4bc1caa1fdbbec31c5bde1c3dfd3638f01c7c33f6a312a6acc2127bf36e"} Jul 29 03:56:27.691093 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.691067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77f89b64c-jk642" event={"ID":"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46","Type":"ContainerStarted","Data":"ff98561436387d706d08ca6fcd786550cddef52eb6764699edbfce3266c8a337"} Jul 29 03:56:27.691188 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.691100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77f89b64c-jk642" event={"ID":"28ed722e-2b3c-4d7e-90ed-5f247f1b0f46","Type":"ContainerStarted","Data":"c6e58b20e9a7013e47a8fdacc9aa12369b0f3b74f178254b3807fa849ea8352b"} Jul 29 03:56:27.693687 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.693647 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65679c454d-bn24p" podStartSLOduration=68.693635318 podStartE2EDuration="1m8.693635318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:27.692928984 +0000 UTC m=+69.985025348" watchObservedRunningTime="2026-07-29 03:56:27.693635318 +0000 UTC m=+69.985731657" Jul 29 03:56:27.714089 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.714031 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-77f89b64c-jk642" podStartSLOduration=44.714009536 podStartE2EDuration="44.714009536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:27.712994121 +0000 UTC m=+70.005090483" watchObservedRunningTime="2026-07-29 03:56:27.714009536 +0000 UTC m=+70.006105899" Jul 29 03:56:27.724231 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.724201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:27.733450 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.733391 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qr5hw" podStartSLOduration=68.144739706 podStartE2EDuration="1m9.733368361s" totalImagesPullingTime="1.588628655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:27.732345424 +0000 UTC m=+70.024441788" watchObservedRunningTime="2026-07-29 03:56:27.733368361 +0000 UTC m=+70.025464723" Jul 29 03:56:27.851678 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.851643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:27.952586 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:27.952505 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:28.015868 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:28.015833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:28.307315 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:28.307236 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:28.309945 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:28.309923 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:28.321920 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:28.321770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:28.418592 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:28.418517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:28.722821 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:28.722787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:28.724268 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:28.724245 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77f89b64c-jk642" Jul 29 03:56:33.524841 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:33.524792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:33.577556 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:33.577434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:35.441920 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:35.441855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:35.512473 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:35.512383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:35.847251 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:35.847123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:36.315151 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:36.315101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:36.376529 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:36.376471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:36.434925 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:36.434869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:38.656593 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.655650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:38.751296 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.751223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:38.771125 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.770607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" event={"ID":"a719ded6-afb9-4ac3-8c4f-1fea94627815","Type":"ContainerStarted","Data":"26fc08d350a8a425f154690b4431675423c55aab743340e9345e2ab4036715f3"} Jul 29 03:56:38.787225 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.787115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2lxw" event={"ID":"fd9e5cff-51ba-49af-9156-5d6714453eed","Type":"ContainerStarted","Data":"8a2c450e96f75f69d07dcf1deac60bc6579da0eef0e928c964bb455f23b75e8d"} Jul 29 03:56:38.789691 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.789345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" event={"ID":"fec72562-75b7-41bc-8a75-86fd188e4023","Type":"ContainerStarted","Data":"c0ddaf66871f57f1ae3bc0761c1b7d5c2192089afdcccc3e4d75cee7c9ac1a9c"} Jul 29 03:56:38.789691 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.789392 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" event={"ID":"fec72562-75b7-41bc-8a75-86fd188e4023","Type":"ContainerStarted","Data":"760f08830bef50ca7f7ca3b13a6f610dba9b945260a8fd43578b3b713b82a92a"} Jul 29 03:56:38.797014 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.796986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6mqc" event={"ID":"525a2c3e-cca1-435f-a3d6-7b94f465d5f5","Type":"ContainerStarted","Data":"6d07a6757e2eb104b36959994496fa6452ed73ae7f8b9f53446cacaf7e48c5f9"} Jul 29 03:56:38.799505 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.799472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" event={"ID":"b1526c58-9b87-4c23-b006-f84bed8f99a1","Type":"ContainerStarted","Data":"e563b9c298578559c08e4e975f5181c2906d91ae7aba5a9faa26c263e0d892a8"} Jul 29 03:56:38.803917 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.803897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4g7" event={"ID":"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7","Type":"ContainerStarted","Data":"549e2e9326f088f52153055ce1dc5d2892a25e41754189ca586c417e218108c6"} Jul 29 03:56:38.806971 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.806827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" event={"ID":"fae15b77-656e-4080-a60b-ced44902b861","Type":"ContainerStarted","Data":"32ef4d36cef39987518686f05330a6fe2ca0d9b2df8ff654c25a9e5ac62a2d9b"} Jul 29 03:56:38.823785 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.823735 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x6mqc" podStartSLOduration=68.492135026 podStartE2EDuration="1m20.82371562s" totalImagesPullingTime="12.331580594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:38.822409966 +0000 UTC m=+81.114506331" watchObservedRunningTime="2026-07-29 03:56:38.82371562 +0000 UTC m=+81.115811982" Jul 29 03:56:38.824689 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.824652 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bj6qs" podStartSLOduration=44.858169869 podStartE2EDuration="55.824640842s" totalImagesPullingTime="10.966470973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:38.797624498 +0000 UTC m=+81.089720865" watchObservedRunningTime="2026-07-29 03:56:38.824640842 +0000 UTC m=+81.116737205" Jul 29 03:56:38.845336 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.845290 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qkfv5" podStartSLOduration=45.572226263 podStartE2EDuration="56.845274105s" totalImagesPullingTime="11.273047842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:38.844602871 +0000 UTC m=+81.136699236" watchObservedRunningTime="2026-07-29 03:56:38.845274105 +0000 UTC m=+81.137370468" Jul 29 03:56:38.869510 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.869449 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-f9gqc" podStartSLOduration=44.726793811 podStartE2EDuration="55.869429927s" totalImagesPullingTime="11.142636116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:38.868390525 +0000 UTC m=+81.160486880" watchObservedRunningTime="2026-07-29 03:56:38.869429927 +0000 UTC m=+81.161526291" Jul 29 03:56:38.886041 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.886000 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79bbc5d5fb-94v2p" podStartSLOduration=1.897937709 podStartE2EDuration="12.885983248s" totalImagesPullingTime="10.988045539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:38.885065114 +0000 UTC m=+81.177161477" watchObservedRunningTime="2026-07-29 03:56:38.885983248 +0000 UTC m=+81.178079611" Jul 29 03:56:38.919966 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:38.919819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:39.021800 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:39.021749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:39.104538 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:39.104503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:39.333347 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:39.333267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:40.043958 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:40.043917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:40.096836 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:40.096796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:42.107937 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:42.107863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:42.206607 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:42.206567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:42.258314 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:42.258278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:42.720873 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:42.720834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:42.784745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:42.784698 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:44.449677 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:44.449635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:44.530953 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:44.530916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:46.264374 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.264292 2576 patch_prober.go:28] interesting pod/image-registry-65679c454d-bn24p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 03:56:46.264374 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.264352 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65679c454d-bn24p" podUID="f49df069-1566-41a1-9349-e85601dd1ffa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:56:46.844183 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.843629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2lxw" event={"ID":"fd9e5cff-51ba-49af-9156-5d6714453eed","Type":"ContainerStarted","Data":"6ddb4fab9b7e5ab0aed78f42b7ff6bc9287f351637c2a024c966f63d5794c31b"} Jul 29 03:56:46.844183 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.843756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n2lxw" Jul 29 03:56:46.846128 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.846095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-n2dwp" event={"ID":"9a404653-f311-4db0-8c78-10e4e9d73000","Type":"ContainerStarted","Data":"bbc87d16ff93460bfc97e65e23053ced1a59c0dbc592d9fe16248883604110a9"} Jul 29 03:56:46.846272 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.846253 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-n2dwp" Jul 29 03:56:46.863165 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.863072 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n2lxw" podStartSLOduration=41.543291503 podStartE2EDuration="52.863053175s" totalImagesPullingTime="11.319761672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:46.862337474 +0000 UTC m=+89.154433839" watchObservedRunningTime="2026-07-29 03:56:46.863053175 +0000 UTC m=+89.155149606" Jul 29 03:56:46.865191 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.865161 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-n2dwp" Jul 29 03:56:46.881080 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:46.880675 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-n2dwp" podStartSLOduration=2.073591305 podStartE2EDuration="20.88065671s" totalImagesPullingTime="18.807065405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:46.878779599 +0000 UTC m=+89.170875985" watchObservedRunningTime="2026-07-29 03:56:46.88065671 +0000 UTC m=+89.172753151" Jul 29 03:56:47.660612 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:47.660522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:48.728145 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:48.728114 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:56:48.885900 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:48.885841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:48.940485 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:48.940407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:50.232935 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.232897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:50.334126 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.334093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:50.459758 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.459725 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:50.499683 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.499618 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cjng5"] Jul 29 03:56:50.519339 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.519318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.521974 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.521950 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 03:56:50.522407 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.522385 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 03:56:50.522499 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.522396 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 03:56:50.522559 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.522497 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 03:56:50.522922 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.522872 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b4bvh\"" Jul 29 03:56:50.546421 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.546394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:50.586333 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-tls\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586523 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-wtmp\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586523 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586523 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-root\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586523 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-sys\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586737 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586737 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-textfile\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586737 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-metrics-client-ca\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.586737 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.586704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59j6m\" (UniqueName: \"kubernetes.io/projected/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-kube-api-access-59j6m\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688023 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.687998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-tls\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688157 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-wtmp\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688157 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688157 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-root\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688157 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-sys\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688349 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:50.688168 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 03:56:50.688349 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688349 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-wtmp\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688349 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-textfile\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688349 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:56:50.688253 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-tls podName:cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f nodeName:}" failed. No retries permitted until 2026-07-29 03:56:51.188237787 +0000 UTC m=+93.480334130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-tls") pod "node-exporter-cjng5" (UID: "cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f") : secret "node-exporter-tls" not found Jul 29 03:56:50.688599 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-sys\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688599 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-root\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688599 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-textfile\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688599 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-metrics-client-ca\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.688599 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.688578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59j6m\" (UniqueName: \"kubernetes.io/projected/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-kube-api-access-59j6m\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.689075 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.689043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-metrics-client-ca\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.689194 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.689173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.691380 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.691357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.697135 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.697089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59j6m\" (UniqueName: \"kubernetes.io/projected/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-kube-api-access-59j6m\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:50.731444 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.731410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:50.855111 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.855039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n2lxw" Jul 29 03:56:50.868029 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.867995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9c4g7" event={"ID":"12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7","Type":"ContainerStarted","Data":"4a08e71f519b501272e35cb14296552d3aa4b883324205dd35fea478fc9e6bd6"} Jul 29 03:56:50.896692 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.896635 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9c4g7" podStartSLOduration=17.161112687 podStartE2EDuration="46.896614662s" totalImagesPullingTime="29.735501975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:56:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:50.894642887 +0000 UTC m=+93.186739277" watchObservedRunningTime="2026-07-29 03:56:50.896614662 +0000 UTC m=+93.188711025" Jul 29 03:56:50.899243 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.899206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:50.994665 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:50.994626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:51.099988 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.099937 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:51.193725 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.193694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-tls\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:51.196489 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.196468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f-node-exporter-tls\") pod \"node-exporter-cjng5\" (UID: \"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f\") " pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:51.198408 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.198372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:51.430535 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.430503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cjng5" Jul 29 03:56:51.462405 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:56:51.462376 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4df7e1_0c63_4ffa_aa1a_9cffd8ba9c1f.slice/crio-a652f0b0b8b220d0b799a4c28683f574cc55cfd4866a1b6b7343806235810602 WatchSource:0}: Error finding container a652f0b0b8b220d0b799a4c28683f574cc55cfd4866a1b6b7343806235810602: Status 404 returned error can't find the container with id a652f0b0b8b220d0b799a4c28683f574cc55cfd4866a1b6b7343806235810602 Jul 29 03:56:51.464694 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.464662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:51.564266 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.564228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:51.874508 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:51.874471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjng5" event={"ID":"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f","Type":"ContainerStarted","Data":"a652f0b0b8b220d0b799a4c28683f574cc55cfd4866a1b6b7343806235810602"} Jul 29 03:56:52.040903 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:52.040843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:52.223632 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:52.221955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:52.318074 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:52.318032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:53.424186 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:53.424155 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65679c454d-bn24p"] Jul 29 03:56:53.449435 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:53.449385 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:53.550371 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:53.550325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:53.633565 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:53.633533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:53.885054 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:53.884975 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f" containerID="c00fd256c489cad03dd8e90eebc0d2c27ab0ad2a63a5a6298d1adae50eebd029" exitCode=0 Jul 29 03:56:53.885054 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:53.885019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjng5" event={"ID":"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f","Type":"ContainerDied","Data":"c00fd256c489cad03dd8e90eebc0d2c27ab0ad2a63a5a6298d1adae50eebd029"} Jul 29 03:56:53.944374 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:53.944345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:54.388211 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:54.388178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:54.575567 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:54.575483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:54.892523 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:54.892484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjng5" event={"ID":"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f","Type":"ContainerStarted","Data":"c89f87c1d68775413030ea7fca75031662280665fb18bafff2708f17af4f36f5"} Jul 29 03:56:54.892705 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:54.892569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjng5" event={"ID":"cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f","Type":"ContainerStarted","Data":"a48f090c9735b8ccff68a9fdeb3fad4767b157b4c3c75f5bd4e419248ea13165"} Jul 29 03:56:54.914965 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:54.914916 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cjng5" podStartSLOduration=3.217692657 podStartE2EDuration="4.914868524s" totalImagesPullingTime="1.596800419s" totalInitContainerRuntime="100.375448ms" isStatefulPod=true podCreationTimestamp="2026-07-29 03:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:54.91345404 +0000 UTC m=+97.205550416" watchObservedRunningTime="2026-07-29 03:56:54.914868524 +0000 UTC m=+97.206964889" Jul 29 03:56:55.280643 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:55.280563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:55.355737 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:55.355699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:55.580770 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:55.580683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:56.822748 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:56.822703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:57.682093 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:57.682055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:57.693064 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:57.693038 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xznfq" Jul 29 03:56:57.783031 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:57.782993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:58.187721 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:58.187693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:56:58.246030 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:56:58.246007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:01.998390 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:01.998347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:02.098760 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:02.098724 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:03.407415 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:03.407383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:03.458940 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:03.458902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:06.635632 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:06.635595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:06.739326 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:06.739293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:06.850852 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:06.850818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:11.089110 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:11.089070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:11.188518 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:11.188484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:12.714432 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:12.714393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:12.820353 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:12.820325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:18.448922 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.448842 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65679c454d-bn24p" podUID="f49df069-1566-41a1-9349-e85601dd1ffa" containerName="registry" containerID="cri-o://e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5" gracePeriod=30 Jul 29 03:57:18.690956 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.690935 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:57:18.799999 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.799934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-trusted-ca\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.799999 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.799968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plh5n\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-kube-api-access-plh5n\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.799999 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.799989 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f49df069-1566-41a1-9349-e85601dd1ffa-ca-trust-extracted\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.800250 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.800033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-registry-certificates\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.800250 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.800056 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-bound-sa-token\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.800250 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.800081 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.800250 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.800104 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-installation-pull-secrets\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.800250 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.800141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-image-registry-private-configuration\") pod \"f49df069-1566-41a1-9349-e85601dd1ffa\" (UID: \"f49df069-1566-41a1-9349-e85601dd1ffa\") " Jul 29 03:57:18.800491 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.800445 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:57:18.800603 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.800575 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:57:18.803001 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.802971 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-kube-api-access-plh5n" (OuterVolumeSpecName: "kube-api-access-plh5n") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "kube-api-access-plh5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:57:18.803131 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.803102 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:57:18.803189 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.803134 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:57:18.803280 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.803258 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:57:18.803318 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.803284 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:57:18.808694 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.808673 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f49df069-1566-41a1-9349-e85601dd1ffa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f49df069-1566-41a1-9349-e85601dd1ffa" (UID: "f49df069-1566-41a1-9349-e85601dd1ffa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:57:18.901356 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901331 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-trusted-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.901356 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901355 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plh5n\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-kube-api-access-plh5n\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.901501 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901369 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f49df069-1566-41a1-9349-e85601dd1ffa-ca-trust-extracted\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.901501 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901382 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f49df069-1566-41a1-9349-e85601dd1ffa-registry-certificates\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.901501 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901395 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-bound-sa-token\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.901501 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901407 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f49df069-1566-41a1-9349-e85601dd1ffa-registry-tls\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.901501 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901419 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-installation-pull-secrets\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.901501 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.901433 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f49df069-1566-41a1-9349-e85601dd1ffa-image-registry-private-configuration\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:57:18.981754 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.981726 2576 generic.go:356] "Generic (PLEG): container finished" podID="f49df069-1566-41a1-9349-e85601dd1ffa" containerID="e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5" exitCode=0 Jul 29 03:57:18.981836 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.981769 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65679c454d-bn24p" Jul 29 03:57:18.981836 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.981798 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65679c454d-bn24p" event={"ID":"f49df069-1566-41a1-9349-e85601dd1ffa","Type":"ContainerDied","Data":"e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5"} Jul 29 03:57:18.981914 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.981840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65679c454d-bn24p" event={"ID":"f49df069-1566-41a1-9349-e85601dd1ffa","Type":"ContainerDied","Data":"cd23d78fb6db2b71b7d6015fd6243c77c18e7d46f609903f25d975048f2ecd50"} Jul 29 03:57:18.981914 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.981862 2576 scope.go:117] "RemoveContainer" containerID="e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5" Jul 29 03:57:18.989854 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.989841 2576 scope.go:117] "RemoveContainer" containerID="e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5" Jul 29 03:57:18.990166 ip-10-0-140-47 kubenswrapper[2576]: E0729 03:57:18.990147 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5\": container with ID starting with e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5 not found: ID does not exist" containerID="e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5" Jul 29 03:57:18.990233 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:18.990178 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5"} err="failed to get container status \"e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5\": rpc error: code = NotFound desc = could not find container \"e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5\": container with ID starting with e2d160d34684ffa6157bb3d4d4c59717a4be47219ee67f966b9b3e2bd00e9fb5 not found: ID does not exist" Jul 29 03:57:19.004602 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:19.004582 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65679c454d-bn24p"] Jul 29 03:57:19.010953 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:19.010932 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65679c454d-bn24p"] Jul 29 03:57:20.069076 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:20.069042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:20.170212 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:20.170184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:20.195645 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:20.195619 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f49df069-1566-41a1-9349-e85601dd1ffa" path="/var/lib/kubelet/pods/f49df069-1566-41a1-9349-e85601dd1ffa/volumes" Jul 29 03:57:20.221243 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:20.221212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:22.191065 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:22.190997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:22.304827 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:22.304797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:24.002003 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:24.001973 2576 generic.go:356] "Generic (PLEG): container finished" podID="596212fb-ef59-4d1a-9770-86fce25ea36b" containerID="9a47659f8e0bd25d897c5056e459b8278216f302ff8274284c8379bb93d0ead9" exitCode=0 Jul 29 03:57:24.002385 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:24.002059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" event={"ID":"596212fb-ef59-4d1a-9770-86fce25ea36b","Type":"ContainerDied","Data":"9a47659f8e0bd25d897c5056e459b8278216f302ff8274284c8379bb93d0ead9"} Jul 29 03:57:24.002385 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:24.002340 2576 scope.go:117] "RemoveContainer" containerID="9a47659f8e0bd25d897c5056e459b8278216f302ff8274284c8379bb93d0ead9" Jul 29 03:57:25.008356 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:25.008327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl" event={"ID":"596212fb-ef59-4d1a-9770-86fce25ea36b","Type":"ContainerStarted","Data":"52b9d1fbdb7225bb446c92a9b049a8a62315256ac9fe1d0830fd543a185a6507"} Jul 29 03:57:27.713438 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:27.713405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:27.812655 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:27.812620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:34.040857 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:34.040824 2576 generic.go:356] "Generic (PLEG): container finished" podID="96e39af8-6a4c-42d1-87d0-850180f27c86" containerID="75fc396a29f8f3073194aab8a7982829544d65d331fe58997396ae7a023384af" exitCode=0 Jul 29 03:57:34.041270 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:34.040864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" event={"ID":"96e39af8-6a4c-42d1-87d0-850180f27c86","Type":"ContainerDied","Data":"75fc396a29f8f3073194aab8a7982829544d65d331fe58997396ae7a023384af"} Jul 29 03:57:34.041270 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:34.041204 2576 scope.go:117] "RemoveContainer" containerID="75fc396a29f8f3073194aab8a7982829544d65d331fe58997396ae7a023384af" Jul 29 03:57:34.660100 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:34.660045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:34.712061 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:34.712026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:35.046307 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:35.046232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rgvpw" event={"ID":"96e39af8-6a4c-42d1-87d0-850180f27c86","Type":"ContainerStarted","Data":"dd720ad00ae99084d4285c835fdd7c5b8770ffcfcfa34c85cc091b80d0167966"} Jul 29 03:57:39.063569 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:39.063531 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49" containerID="6e07f3190d0b28216f2211a869cf74210b5eaac6c9e9f9ffd6b1518f6de688ce" exitCode=0 Jul 29 03:57:39.063931 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:39.063602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rhf5t" event={"ID":"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49","Type":"ContainerDied","Data":"6e07f3190d0b28216f2211a869cf74210b5eaac6c9e9f9ffd6b1518f6de688ce"} Jul 29 03:57:39.063995 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:39.063939 2576 scope.go:117] "RemoveContainer" containerID="6e07f3190d0b28216f2211a869cf74210b5eaac6c9e9f9ffd6b1518f6de688ce" Jul 29 03:57:39.833595 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:39.833562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:40.069052 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:40.069019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rhf5t" event={"ID":"3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49","Type":"ContainerStarted","Data":"3b647198281cee67dc819cb3971af9eb96bdbf5ee85ad10043101eda46c1bf5e"} Jul 29 03:57:42.697917 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:42.697860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:57:42.756148 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:57:42.756104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:09.720062 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:09.719964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:09.820257 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:09.820226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:14.784703 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.784671 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c47c9867f-g6fvr"] Jul 29 03:58:14.785104 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.784947 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f49df069-1566-41a1-9349-e85601dd1ffa" containerName="registry" Jul 29 03:58:14.785104 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.784959 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f49df069-1566-41a1-9349-e85601dd1ffa" containerName="registry" Jul 29 03:58:14.785104 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.785013 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f49df069-1566-41a1-9349-e85601dd1ffa" containerName="registry" Jul 29 03:58:14.786947 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.786925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.789551 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.789525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 03:58:14.789665 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.789534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 03:58:14.789665 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.789594 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rqf8r\"" Jul 29 03:58:14.789665 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.789600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 03:58:14.789665 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.789632 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 03:58:14.789665 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.789564 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 03:58:14.794555 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.794537 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 03:58:14.804344 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.804322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c47c9867f-g6fvr"] Jul 29 03:58:14.881814 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.881788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-metrics-client-ca\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.881954 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.881824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.881954 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.881916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.882040 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.881970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-telemeter-client-tls\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.882040 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.882000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-serving-certs-ca-bundle\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.882040 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.882016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9z7\" (UniqueName: \"kubernetes.io/projected/98231da0-b5d1-473a-9721-f43f769935ae-kube-api-access-hs9z7\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.882154 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.882054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-federate-client-tls\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.882154 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.882077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-secret-telemeter-client\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.982697 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.982673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.982820 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.982718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-telemeter-client-tls\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.982820 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.982749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-serving-certs-ca-bundle\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.982967 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.982870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9z7\" (UniqueName: \"kubernetes.io/projected/98231da0-b5d1-473a-9721-f43f769935ae-kube-api-access-hs9z7\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.982967 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.982946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-federate-client-tls\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.983077 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.982973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-secret-telemeter-client\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.983077 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.983020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-metrics-client-ca\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.983077 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.983049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.983509 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.983483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-serving-certs-ca-bundle\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.983811 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.983790 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-metrics-client-ca\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.984292 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.984267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98231da0-b5d1-473a-9721-f43f769935ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.985616 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.985598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-federate-client-tls\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.985748 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.985727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-secret-telemeter-client\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.985867 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.985853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.985944 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.985924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98231da0-b5d1-473a-9721-f43f769935ae-telemeter-client-tls\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:14.991670 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:14.991646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9z7\" (UniqueName: \"kubernetes.io/projected/98231da0-b5d1-473a-9721-f43f769935ae-kube-api-access-hs9z7\") pod \"telemeter-client-c47c9867f-g6fvr\" (UID: \"98231da0-b5d1-473a-9721-f43f769935ae\") " pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:15.096522 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:15.096470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" Jul 29 03:58:15.230071 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:15.230044 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c47c9867f-g6fvr"] Jul 29 03:58:15.233654 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:58:15.233618 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98231da0_b5d1_473a_9721_f43f769935ae.slice/crio-16f92ca362f1f41fb478e98573fb707c267719ba77ae3ffe23926b0b1c4c0944 WatchSource:0}: Error finding container 16f92ca362f1f41fb478e98573fb707c267719ba77ae3ffe23926b0b1c4c0944: Status 404 returned error can't find the container with id 16f92ca362f1f41fb478e98573fb707c267719ba77ae3ffe23926b0b1c4c0944 Jul 29 03:58:16.206927 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:16.206866 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" event={"ID":"98231da0-b5d1-473a-9721-f43f769935ae","Type":"ContainerStarted","Data":"16f92ca362f1f41fb478e98573fb707c267719ba77ae3ffe23926b0b1c4c0944"} Jul 29 03:58:17.212262 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:17.212221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" event={"ID":"98231da0-b5d1-473a-9721-f43f769935ae","Type":"ContainerStarted","Data":"4af096b01d038573bc86f7f66dcbcbcc83fe680c8af5b2fd2ec8551e70bf6b53"} Jul 29 03:58:18.217732 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:18.217651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" event={"ID":"98231da0-b5d1-473a-9721-f43f769935ae","Type":"ContainerStarted","Data":"a1e534fdc2608e70f02a645ca1ec19809e2680c962559d0f5a835bf76987bae9"} Jul 29 03:58:18.217732 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:18.217708 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" event={"ID":"98231da0-b5d1-473a-9721-f43f769935ae","Type":"ContainerStarted","Data":"6a73b29b61dc64e881969ce415c562b4154714b64bf36c2a7986cc46194eb486"} Jul 29 03:58:18.240047 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:18.239996 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c47c9867f-g6fvr" podStartSLOduration=1.600064326 podStartE2EDuration="4.239979046s" totalImagesPullingTime="2.63991472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:58:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:58:18.237500667 +0000 UTC m=+180.529597029" watchObservedRunningTime="2026-07-29 03:58:18.239979046 +0000 UTC m=+180.532075409" Jul 29 03:58:19.126023 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:19.125989 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:19.226318 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:19.226283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:19.325848 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:19.325821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:19.858849 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:19.858816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:19.958681 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:19.958648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:20.741646 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:20.741610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:24.920864 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:24.920832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:24.978916 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:24.978864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:36.847705 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:36.847663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:36.901987 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:36.901958 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:40.034903 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:40.034833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:40.135716 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:40.135683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:40.191478 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:40.191444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:42.704374 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:42.704340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:58:42.805023 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:58:42.804984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:08.872677 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:08.872629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:08.923373 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:08.923343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:09.581874 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.581840 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54"] Jul 29 03:59:09.585101 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.585086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.588620 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.588598 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 03:59:09.589745 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.589729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 03:59:09.589823 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.589742 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jjbqn\"" Jul 29 03:59:09.595341 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.595318 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54"] Jul 29 03:59:09.688353 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.688322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.688491 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.688398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.688491 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.688424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnrq\" (UniqueName: \"kubernetes.io/projected/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-kube-api-access-tqnrq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.789737 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.789706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.789869 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.789746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnrq\" (UniqueName: \"kubernetes.io/projected/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-kube-api-access-tqnrq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.789869 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.789775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.790187 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.790167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.790227 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.790182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.797619 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.797596 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnrq\" (UniqueName: \"kubernetes.io/projected/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-kube-api-access-tqnrq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.894281 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.894262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:09.971278 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:09.971227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:10.022935 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:10.022846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54"] Jul 29 03:59:10.025105 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:59:10.025082 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb091ed_8532_4c94_a4b7_9d26ef8c01c9.slice/crio-f45ad0124789a381287972b980a0d7a927741610ba7a0b5a45b5598def01dcb8 WatchSource:0}: Error finding container f45ad0124789a381287972b980a0d7a927741610ba7a0b5a45b5598def01dcb8: Status 404 returned error can't find the container with id f45ad0124789a381287972b980a0d7a927741610ba7a0b5a45b5598def01dcb8 Jul 29 03:59:10.405927 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:10.405897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" event={"ID":"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9","Type":"ContainerStarted","Data":"f45ad0124789a381287972b980a0d7a927741610ba7a0b5a45b5598def01dcb8"} Jul 29 03:59:14.208652 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:14.208615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:14.256739 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:14.256694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:17.434398 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:17.434366 2576 generic.go:356] "Generic (PLEG): container finished" podID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerID="da746be4e766cf341ec3a322f5a30ae7bcb2acaae73724f713e675b88b985e62" exitCode=0 Jul 29 03:59:17.434830 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:17.434443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" event={"ID":"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9","Type":"ContainerDied","Data":"da746be4e766cf341ec3a322f5a30ae7bcb2acaae73724f713e675b88b985e62"} Jul 29 03:59:20.448805 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:20.448768 2576 generic.go:356] "Generic (PLEG): container finished" podID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerID="6fcf8f2434790e0db0c90fd53aef91379eeca0d00b37f6239305f66f31a5ae84" exitCode=0 Jul 29 03:59:20.449196 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:20.448818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" event={"ID":"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9","Type":"ContainerDied","Data":"6fcf8f2434790e0db0c90fd53aef91379eeca0d00b37f6239305f66f31a5ae84"} Jul 29 03:59:27.490104 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:27.490073 2576 generic.go:356] "Generic (PLEG): container finished" podID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerID="43bef35a33cefba1c5e94645c78026a028bb0ec475a0c065de7332f00f0775ab" exitCode=0 Jul 29 03:59:27.490481 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:27.490153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" event={"ID":"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9","Type":"ContainerDied","Data":"43bef35a33cefba1c5e94645c78026a028bb0ec475a0c065de7332f00f0775ab"} Jul 29 03:59:28.621737 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.621715 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:28.749750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.749664 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-util\") pod \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " Jul 29 03:59:28.749750 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.749726 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-bundle\") pod \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " Jul 29 03:59:28.749987 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.749772 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqnrq\" (UniqueName: \"kubernetes.io/projected/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-kube-api-access-tqnrq\") pod \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\" (UID: \"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9\") " Jul 29 03:59:28.750405 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.750382 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-bundle" (OuterVolumeSpecName: "bundle") pod "9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" (UID: "9bb091ed-8532-4c94-a4b7-9d26ef8c01c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:59:28.752147 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.752124 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-kube-api-access-tqnrq" (OuterVolumeSpecName: "kube-api-access-tqnrq") pod "9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" (UID: "9bb091ed-8532-4c94-a4b7-9d26ef8c01c9"). InnerVolumeSpecName "kube-api-access-tqnrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:59:28.755108 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.755085 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-util" (OuterVolumeSpecName: "util") pod "9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" (UID: "9bb091ed-8532-4c94-a4b7-9d26ef8c01c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:59:28.851199 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.851155 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-util\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:28.851199 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.851195 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-bundle\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:28.851199 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:28.851209 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnrq\" (UniqueName: \"kubernetes.io/projected/9bb091ed-8532-4c94-a4b7-9d26ef8c01c9-kube-api-access-tqnrq\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:29.499787 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:29.499761 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" Jul 29 03:59:29.499977 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:29.499754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w8r54" event={"ID":"9bb091ed-8532-4c94-a4b7-9d26ef8c01c9","Type":"ContainerDied","Data":"f45ad0124789a381287972b980a0d7a927741610ba7a0b5a45b5598def01dcb8"} Jul 29 03:59:29.499977 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:29.499904 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f45ad0124789a381287972b980a0d7a927741610ba7a0b5a45b5598def01dcb8" Jul 29 03:59:30.419263 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:30.419231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:31.088780 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:31.088741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:31.865420 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:31.865381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:32.391865 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.391835 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj"] Jul 29 03:59:32.392140 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.392127 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerName="pull" Jul 29 03:59:32.392191 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.392142 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerName="pull" Jul 29 03:59:32.392191 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.392157 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerName="util" Jul 29 03:59:32.392191 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.392165 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerName="util" Jul 29 03:59:32.392191 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.392179 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerName="extract" Jul 29 03:59:32.392191 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.392184 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerName="extract" Jul 29 03:59:32.392342 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.392259 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb091ed-8532-4c94-a4b7-9d26ef8c01c9" containerName="extract" Jul 29 03:59:32.433370 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.433344 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj"] Jul 29 03:59:32.433505 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.433433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.436098 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.436076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 29 03:59:32.436349 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.436336 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-b5p6s\"" Jul 29 03:59:32.436508 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.436492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 29 03:59:32.579661 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.579636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05fe62c6-347a-4b1b-a9f2-a86ce1a7976e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5tnnj\" (UID: \"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.579774 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.579681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfz86\" (UniqueName: \"kubernetes.io/projected/05fe62c6-347a-4b1b-a9f2-a86ce1a7976e-kube-api-access-bfz86\") pod \"cert-manager-operator-controller-manager-55c444bf86-5tnnj\" (UID: \"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.680544 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.680483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05fe62c6-347a-4b1b-a9f2-a86ce1a7976e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5tnnj\" (UID: \"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.680544 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.680535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfz86\" (UniqueName: \"kubernetes.io/projected/05fe62c6-347a-4b1b-a9f2-a86ce1a7976e-kube-api-access-bfz86\") pod \"cert-manager-operator-controller-manager-55c444bf86-5tnnj\" (UID: \"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.680818 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.680799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05fe62c6-347a-4b1b-a9f2-a86ce1a7976e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5tnnj\" (UID: \"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.689000 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.688979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfz86\" (UniqueName: \"kubernetes.io/projected/05fe62c6-347a-4b1b-a9f2-a86ce1a7976e-kube-api-access-bfz86\") pod \"cert-manager-operator-controller-manager-55c444bf86-5tnnj\" (UID: \"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.742379 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.742360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" Jul 29 03:59:32.900125 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:32.900101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj"] Jul 29 03:59:32.901664 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:59:32.901637 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fe62c6_347a_4b1b_a9f2_a86ce1a7976e.slice/crio-fa15b46a27dbfa44a7cca82a753d240adb508899bd66616eb538c243d3519e4c WatchSource:0}: Error finding container fa15b46a27dbfa44a7cca82a753d240adb508899bd66616eb538c243d3519e4c: Status 404 returned error can't find the container with id fa15b46a27dbfa44a7cca82a753d240adb508899bd66616eb538c243d3519e4c Jul 29 03:59:33.517194 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:33.517154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" event={"ID":"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e","Type":"ContainerStarted","Data":"fa15b46a27dbfa44a7cca82a753d240adb508899bd66616eb538c243d3519e4c"} Jul 29 03:59:34.042446 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:34.042414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:34.142832 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:34.142794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:35.527282 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:35.527247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" event={"ID":"05fe62c6-347a-4b1b-a9f2-a86ce1a7976e","Type":"ContainerStarted","Data":"aaabfda55b235b759777b281146a29826d0a3c40480e5a1054bf942e51c3a172"} Jul 29 03:59:35.549852 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:35.549794 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5tnnj" podStartSLOduration=1.5751729509999999 podStartE2EDuration="3.549778988s" totalImagesPullingTime="1.974606037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:59:35.547896257 +0000 UTC m=+257.839992612" watchObservedRunningTime="2026-07-29 03:59:35.549778988 +0000 UTC m=+257.841875351" Jul 29 03:59:35.931683 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:35.931639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:36.038344 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.038305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:36.151528 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.151497 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v9szh"] Jul 29 03:59:36.170182 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.170158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.172040 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.172019 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v9szh"] Jul 29 03:59:36.172835 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.172803 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 03:59:36.172954 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.172922 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 03:59:36.310497 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.310424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4ff9019-1abb-4a5f-908f-b254a7484938-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v9szh\" (UID: \"c4ff9019-1abb-4a5f-908f-b254a7484938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.310497 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.310464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkc2r\" (UniqueName: \"kubernetes.io/projected/c4ff9019-1abb-4a5f-908f-b254a7484938-kube-api-access-fkc2r\") pod \"cert-manager-webhook-6486ffd997-v9szh\" (UID: \"c4ff9019-1abb-4a5f-908f-b254a7484938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.410792 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.410766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4ff9019-1abb-4a5f-908f-b254a7484938-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v9szh\" (UID: \"c4ff9019-1abb-4a5f-908f-b254a7484938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.410952 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.410807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkc2r\" (UniqueName: \"kubernetes.io/projected/c4ff9019-1abb-4a5f-908f-b254a7484938-kube-api-access-fkc2r\") pod \"cert-manager-webhook-6486ffd997-v9szh\" (UID: \"c4ff9019-1abb-4a5f-908f-b254a7484938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.420388 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.420362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4ff9019-1abb-4a5f-908f-b254a7484938-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v9szh\" (UID: \"c4ff9019-1abb-4a5f-908f-b254a7484938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.420551 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.420531 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkc2r\" (UniqueName: \"kubernetes.io/projected/c4ff9019-1abb-4a5f-908f-b254a7484938-kube-api-access-fkc2r\") pod \"cert-manager-webhook-6486ffd997-v9szh\" (UID: \"c4ff9019-1abb-4a5f-908f-b254a7484938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.479861 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.479839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:36.632644 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.632623 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v9szh"] Jul 29 03:59:36.635145 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:59:36.635116 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4ff9019_1abb_4a5f_908f_b254a7484938.slice/crio-a9c1684e6f6e0007cbe5a50a042045ab55a01f4fbdefca2bcdfc17b19ba84c7d WatchSource:0}: Error finding container a9c1684e6f6e0007cbe5a50a042045ab55a01f4fbdefca2bcdfc17b19ba84c7d: Status 404 returned error can't find the container with id a9c1684e6f6e0007cbe5a50a042045ab55a01f4fbdefca2bcdfc17b19ba84c7d Jul 29 03:59:36.851902 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.851827 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96"] Jul 29 03:59:36.886405 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.886381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96"] Jul 29 03:59:36.886521 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.886469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:36.889150 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.889128 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 03:59:36.889256 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.889136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jjbqn\"" Jul 29 03:59:36.889256 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:36.889160 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 03:59:37.015553 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.015525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxg4d\" (UniqueName: \"kubernetes.io/projected/d342375a-254f-43c6-b235-e1b64d90f728-kube-api-access-wxg4d\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.015676 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.015561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.015676 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.015588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.115979 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.115919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxg4d\" (UniqueName: \"kubernetes.io/projected/d342375a-254f-43c6-b235-e1b64d90f728-kube-api-access-wxg4d\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.115979 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.115960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.116125 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.115990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.116286 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.116268 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.116325 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.116302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.125842 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.125824 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxg4d\" (UniqueName: \"kubernetes.io/projected/d342375a-254f-43c6-b235-e1b64d90f728-kube-api-access-wxg4d\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.196087 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.196067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:37.323965 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.323942 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96"] Jul 29 03:59:37.326025 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:59:37.326001 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd342375a_254f_43c6_b235_e1b64d90f728.slice/crio-91d1ffc1e15e8b92b3e6b846e8cc01c80b9ca093740613018e312718347598ef WatchSource:0}: Error finding container 91d1ffc1e15e8b92b3e6b846e8cc01c80b9ca093740613018e312718347598ef: Status 404 returned error can't find the container with id 91d1ffc1e15e8b92b3e6b846e8cc01c80b9ca093740613018e312718347598ef Jul 29 03:59:37.538755 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.538722 2576 generic.go:356] "Generic (PLEG): container finished" podID="d342375a-254f-43c6-b235-e1b64d90f728" containerID="9bf97ee543408e542c4b06eab2b106fbc64e94566dc4b6991ad7df67baa39fb8" exitCode=0 Jul 29 03:59:37.538927 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.538808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" event={"ID":"d342375a-254f-43c6-b235-e1b64d90f728","Type":"ContainerDied","Data":"9bf97ee543408e542c4b06eab2b106fbc64e94566dc4b6991ad7df67baa39fb8"} Jul 29 03:59:37.538927 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.538857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" event={"ID":"d342375a-254f-43c6-b235-e1b64d90f728","Type":"ContainerStarted","Data":"91d1ffc1e15e8b92b3e6b846e8cc01c80b9ca093740613018e312718347598ef"} Jul 29 03:59:37.540126 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:37.540096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" event={"ID":"c4ff9019-1abb-4a5f-908f-b254a7484938","Type":"ContainerStarted","Data":"a9c1684e6f6e0007cbe5a50a042045ab55a01f4fbdefca2bcdfc17b19ba84c7d"} Jul 29 03:59:38.291686 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:38.291652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:38.496014 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:38.495921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:40.560553 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:40.560460 2576 generic.go:356] "Generic (PLEG): container finished" podID="d342375a-254f-43c6-b235-e1b64d90f728" containerID="88d94dd525793abed7058a73a7dac8a5e625764613795b819c888a46bc3fc72e" exitCode=0 Jul 29 03:59:40.560973 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:40.560558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" event={"ID":"d342375a-254f-43c6-b235-e1b64d90f728","Type":"ContainerDied","Data":"88d94dd525793abed7058a73a7dac8a5e625764613795b819c888a46bc3fc72e"} Jul 29 03:59:40.562198 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:40.562169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" event={"ID":"c4ff9019-1abb-4a5f-908f-b254a7484938","Type":"ContainerStarted","Data":"dc4d29a5f6d74dca0677c4c718b506133b9d5d99d1aab03b5b10f8cbe7b91087"} Jul 29 03:59:40.562323 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:40.562299 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:40.625254 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:40.625210 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" podStartSLOduration=1.011714413 podStartE2EDuration="4.625192881s" totalImagesPullingTime="3.613478468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:59:40.624615838 +0000 UTC m=+262.916712203" watchObservedRunningTime="2026-07-29 03:59:40.625192881 +0000 UTC m=+262.917289242" Jul 29 03:59:41.034278 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:41.034245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:41.086154 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:41.086124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:41.568993 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:41.568950 2576 generic.go:356] "Generic (PLEG): container finished" podID="d342375a-254f-43c6-b235-e1b64d90f728" containerID="8fb54eef7610895f6fbd4fd3de71e7725af05dc845d46a9864be9fa3981ef0e2" exitCode=0 Jul 29 03:59:41.569387 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:41.569012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" event={"ID":"d342375a-254f-43c6-b235-e1b64d90f728","Type":"ContainerDied","Data":"8fb54eef7610895f6fbd4fd3de71e7725af05dc845d46a9864be9fa3981ef0e2"} Jul 29 03:59:42.694084 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.694062 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:42.863898 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.863808 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-util\") pod \"d342375a-254f-43c6-b235-e1b64d90f728\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " Jul 29 03:59:42.863898 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.863850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-bundle\") pod \"d342375a-254f-43c6-b235-e1b64d90f728\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " Jul 29 03:59:42.864110 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.863912 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxg4d\" (UniqueName: \"kubernetes.io/projected/d342375a-254f-43c6-b235-e1b64d90f728-kube-api-access-wxg4d\") pod \"d342375a-254f-43c6-b235-e1b64d90f728\" (UID: \"d342375a-254f-43c6-b235-e1b64d90f728\") " Jul 29 03:59:42.864317 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.864294 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-bundle" (OuterVolumeSpecName: "bundle") pod "d342375a-254f-43c6-b235-e1b64d90f728" (UID: "d342375a-254f-43c6-b235-e1b64d90f728"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:59:42.866160 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.866131 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d342375a-254f-43c6-b235-e1b64d90f728-kube-api-access-wxg4d" (OuterVolumeSpecName: "kube-api-access-wxg4d") pod "d342375a-254f-43c6-b235-e1b64d90f728" (UID: "d342375a-254f-43c6-b235-e1b64d90f728"). InnerVolumeSpecName "kube-api-access-wxg4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:59:42.868094 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.868070 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-util" (OuterVolumeSpecName: "util") pod "d342375a-254f-43c6-b235-e1b64d90f728" (UID: "d342375a-254f-43c6-b235-e1b64d90f728"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:59:42.965024 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.964995 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxg4d\" (UniqueName: \"kubernetes.io/projected/d342375a-254f-43c6-b235-e1b64d90f728-kube-api-access-wxg4d\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:42.965024 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.965020 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-util\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:42.965158 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:42.965033 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d342375a-254f-43c6-b235-e1b64d90f728-bundle\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:43.580794 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:43.580754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" event={"ID":"d342375a-254f-43c6-b235-e1b64d90f728","Type":"ContainerDied","Data":"91d1ffc1e15e8b92b3e6b846e8cc01c80b9ca093740613018e312718347598ef"} Jul 29 03:59:43.580794 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:43.580781 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frsq96" Jul 29 03:59:43.580794 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:43.580811 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d1ffc1e15e8b92b3e6b846e8cc01c80b9ca093740613018e312718347598ef" Jul 29 03:59:43.730085 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:43.730063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:43.831088 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:43.831023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:45.493893 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:45.493861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:46.571328 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:46.571301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v9szh" Jul 29 03:59:46.844834 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:46.844756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:46.944179 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:46.944146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:52.709149 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:52.709106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:52.809839 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:52.809804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:54.903100 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903070 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2"] Jul 29 03:59:54.903459 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903353 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d342375a-254f-43c6-b235-e1b64d90f728" containerName="util" Jul 29 03:59:54.903459 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903364 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d342375a-254f-43c6-b235-e1b64d90f728" containerName="util" Jul 29 03:59:54.903459 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903382 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d342375a-254f-43c6-b235-e1b64d90f728" containerName="extract" Jul 29 03:59:54.903459 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903388 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d342375a-254f-43c6-b235-e1b64d90f728" containerName="extract" Jul 29 03:59:54.903459 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903398 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d342375a-254f-43c6-b235-e1b64d90f728" containerName="pull" Jul 29 03:59:54.903459 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903403 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d342375a-254f-43c6-b235-e1b64d90f728" containerName="pull" Jul 29 03:59:54.903663 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.903476 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d342375a-254f-43c6-b235-e1b64d90f728" containerName="extract" Jul 29 03:59:54.906728 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.906711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:54.909227 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.909206 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jjbqn\"" Jul 29 03:59:54.909347 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.909251 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 03:59:54.909347 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.909251 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 03:59:54.914310 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:54.914284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2"] Jul 29 03:59:55.066860 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.066831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.067026 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.066871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ff5\" (UniqueName: \"kubernetes.io/projected/75c4af0b-f21e-4963-ac82-a9b2b25533b4-kube-api-access-m5ff5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.067026 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.066950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.084542 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.084464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 03:59:55.167854 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.167770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.167854 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.167824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ff5\" (UniqueName: \"kubernetes.io/projected/75c4af0b-f21e-4963-ac82-a9b2b25533b4-kube-api-access-m5ff5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.167854 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.167856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.168185 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.168165 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.168240 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.168196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.176587 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.176560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ff5\" (UniqueName: \"kubernetes.io/projected/75c4af0b-f21e-4963-ac82-a9b2b25533b4-kube-api-access-m5ff5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.216679 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.216653 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:55.345386 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.345359 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2"] Jul 29 03:59:55.348983 ip-10-0-140-47 kubenswrapper[2576]: W0729 03:59:55.348958 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c4af0b_f21e_4963_ac82_a9b2b25533b4.slice/crio-d2cca77d038f14d1dbb8eaaf762b478edd87d9aece1d7dc3a090a6d1e7f663f8 WatchSource:0}: Error finding container d2cca77d038f14d1dbb8eaaf762b478edd87d9aece1d7dc3a090a6d1e7f663f8: Status 404 returned error can't find the container with id d2cca77d038f14d1dbb8eaaf762b478edd87d9aece1d7dc3a090a6d1e7f663f8 Jul 29 03:59:55.632363 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.632330 2576 generic.go:356] "Generic (PLEG): container finished" podID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerID="0a55267573df6b2f5ae966eae60a3413e0a931b5e0589fa14d04330b53e7c35b" exitCode=0 Jul 29 03:59:55.632533 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.632386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" event={"ID":"75c4af0b-f21e-4963-ac82-a9b2b25533b4","Type":"ContainerDied","Data":"0a55267573df6b2f5ae966eae60a3413e0a931b5e0589fa14d04330b53e7c35b"} Jul 29 03:59:55.632533 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:55.632430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" event={"ID":"75c4af0b-f21e-4963-ac82-a9b2b25533b4","Type":"ContainerStarted","Data":"d2cca77d038f14d1dbb8eaaf762b478edd87d9aece1d7dc3a090a6d1e7f663f8"} Jul 29 03:59:56.638812 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:56.638779 2576 generic.go:356] "Generic (PLEG): container finished" podID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerID="40dd2ac8b3af271f3ba3a72ddfca6c6b8bc7921efea3fd323cf9741f3084f1da" exitCode=0 Jul 29 03:59:56.639201 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:56.638848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" event={"ID":"75c4af0b-f21e-4963-ac82-a9b2b25533b4","Type":"ContainerDied","Data":"40dd2ac8b3af271f3ba3a72ddfca6c6b8bc7921efea3fd323cf9741f3084f1da"} Jul 29 03:59:57.645426 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:57.645395 2576 generic.go:356] "Generic (PLEG): container finished" podID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerID="fbcfb0edc1190253f5061daf77d39beee028917110161da60eb3087652ea7131" exitCode=0 Jul 29 03:59:57.645812 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:57.645484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" event={"ID":"75c4af0b-f21e-4963-ac82-a9b2b25533b4","Type":"ContainerDied","Data":"fbcfb0edc1190253f5061daf77d39beee028917110161da60eb3087652ea7131"} Jul 29 03:59:58.775977 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.775948 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:58.897392 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.897364 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-util\") pod \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " Jul 29 03:59:58.897535 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.897399 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5ff5\" (UniqueName: \"kubernetes.io/projected/75c4af0b-f21e-4963-ac82-a9b2b25533b4-kube-api-access-m5ff5\") pod \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " Jul 29 03:59:58.897535 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.897456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-bundle\") pod \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\" (UID: \"75c4af0b-f21e-4963-ac82-a9b2b25533b4\") " Jul 29 03:59:58.898205 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.898172 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-bundle" (OuterVolumeSpecName: "bundle") pod "75c4af0b-f21e-4963-ac82-a9b2b25533b4" (UID: "75c4af0b-f21e-4963-ac82-a9b2b25533b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:59:58.899658 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.899626 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c4af0b-f21e-4963-ac82-a9b2b25533b4-kube-api-access-m5ff5" (OuterVolumeSpecName: "kube-api-access-m5ff5") pod "75c4af0b-f21e-4963-ac82-a9b2b25533b4" (UID: "75c4af0b-f21e-4963-ac82-a9b2b25533b4"). InnerVolumeSpecName "kube-api-access-m5ff5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:59:58.903348 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.903291 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-util" (OuterVolumeSpecName: "util") pod "75c4af0b-f21e-4963-ac82-a9b2b25533b4" (UID: "75c4af0b-f21e-4963-ac82-a9b2b25533b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:59:58.998054 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.998030 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-bundle\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:58.998054 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.998050 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c4af0b-f21e-4963-ac82-a9b2b25533b4-util\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:58.998187 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:58.998071 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ff5\" (UniqueName: \"kubernetes.io/projected/75c4af0b-f21e-4963-ac82-a9b2b25533b4-kube-api-access-m5ff5\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 03:59:59.656700 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:59.656673 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" Jul 29 03:59:59.656869 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:59.656704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvvn2" event={"ID":"75c4af0b-f21e-4963-ac82-a9b2b25533b4","Type":"ContainerDied","Data":"d2cca77d038f14d1dbb8eaaf762b478edd87d9aece1d7dc3a090a6d1e7f663f8"} Jul 29 03:59:59.656869 ip-10-0-140-47 kubenswrapper[2576]: I0729 03:59:59.656745 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2cca77d038f14d1dbb8eaaf762b478edd87d9aece1d7dc3a090a6d1e7f663f8" Jul 29 04:00:05.322730 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:05.322697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:05.468453 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:05.468423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:08.631712 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:08.631673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:08.735386 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:08.735346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:09.302700 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.302654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:09.542097 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.542058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:09.652810 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.652774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:09.682591 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682565 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv"] Jul 29 04:00:09.682846 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682834 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerName="util" Jul 29 04:00:09.682906 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682848 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerName="util" Jul 29 04:00:09.682906 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682857 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerName="pull" Jul 29 04:00:09.682906 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682862 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerName="pull" Jul 29 04:00:09.682906 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682904 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerName="extract" Jul 29 04:00:09.683028 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682913 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerName="extract" Jul 29 04:00:09.683028 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.682957 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="75c4af0b-f21e-4963-ac82-a9b2b25533b4" containerName="extract" Jul 29 04:00:09.699450 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.699420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.702303 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.702280 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 04:00:09.702487 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.702302 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 04:00:09.702560 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.702340 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 04:00:09.703037 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.703017 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 04:00:09.703137 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.703043 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-npvqn\"" Jul 29 04:00:09.704499 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.704467 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv"] Jul 29 04:00:09.767841 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.767815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edc39eb8-388c-4fce-a9da-3281b04150c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.768027 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.767869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edc39eb8-388c-4fce-a9da-3281b04150c7-tmp\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.768027 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.767927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edc39eb8-388c-4fce-a9da-3281b04150c7-manifests\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.768027 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.767951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw2w\" (UniqueName: \"kubernetes.io/projected/edc39eb8-388c-4fce-a9da-3281b04150c7-kube-api-access-xpw2w\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.768027 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.767972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc39eb8-388c-4fce-a9da-3281b04150c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.868972 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.868946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edc39eb8-388c-4fce-a9da-3281b04150c7-tmp\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.869124 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.868979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edc39eb8-388c-4fce-a9da-3281b04150c7-manifests\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.869124 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.868998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw2w\" (UniqueName: \"kubernetes.io/projected/edc39eb8-388c-4fce-a9da-3281b04150c7-kube-api-access-xpw2w\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.869124 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.869111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc39eb8-388c-4fce-a9da-3281b04150c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.869269 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.869162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edc39eb8-388c-4fce-a9da-3281b04150c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.869377 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.869357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edc39eb8-388c-4fce-a9da-3281b04150c7-manifests\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.871359 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.871328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edc39eb8-388c-4fce-a9da-3281b04150c7-tmp\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.871539 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.871522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edc39eb8-388c-4fce-a9da-3281b04150c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.871609 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.871592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc39eb8-388c-4fce-a9da-3281b04150c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:09.876924 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:09.876904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw2w\" (UniqueName: \"kubernetes.io/projected/edc39eb8-388c-4fce-a9da-3281b04150c7-kube-api-access-xpw2w\") pod \"opendatahub-operator-controller-manager-d56969f4b-fbcdv\" (UID: \"edc39eb8-388c-4fce-a9da-3281b04150c7\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:10.011771 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:10.011685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:10.180876 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:10.180854 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv"] Jul 29 04:00:10.183525 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:00:10.183496 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc39eb8_388c_4fce_a9da_3281b04150c7.slice/crio-54d85b049b02a52b63d41db4e18f853a3be2a11f721b4fbe80c8081338c65f1f WatchSource:0}: Error finding container 54d85b049b02a52b63d41db4e18f853a3be2a11f721b4fbe80c8081338c65f1f: Status 404 returned error can't find the container with id 54d85b049b02a52b63d41db4e18f853a3be2a11f721b4fbe80c8081338c65f1f Jul 29 04:00:10.358226 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:10.358133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:10.462673 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:10.462645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:10.704598 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:10.704560 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" event={"ID":"edc39eb8-388c-4fce-a9da-3281b04150c7","Type":"ContainerStarted","Data":"54d85b049b02a52b63d41db4e18f853a3be2a11f721b4fbe80c8081338c65f1f"} Jul 29 04:00:13.721103 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:13.721062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" event={"ID":"edc39eb8-388c-4fce-a9da-3281b04150c7","Type":"ContainerStarted","Data":"9d60320c03e2cfed7f8b59dfddec23a6cd4c2e7db33ff6b78beda7850b135b12"} Jul 29 04:00:14.726749 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:14.726717 2576 generic.go:356] "Generic (PLEG): container finished" podID="edc39eb8-388c-4fce-a9da-3281b04150c7" containerID="9d60320c03e2cfed7f8b59dfddec23a6cd4c2e7db33ff6b78beda7850b135b12" exitCode=0 Jul 29 04:00:14.727242 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:14.726785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" event={"ID":"edc39eb8-388c-4fce-a9da-3281b04150c7","Type":"ContainerDied","Data":"9d60320c03e2cfed7f8b59dfddec23a6cd4c2e7db33ff6b78beda7850b135b12"} Jul 29 04:00:15.732875 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:15.732839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" event={"ID":"edc39eb8-388c-4fce-a9da-3281b04150c7","Type":"ContainerStarted","Data":"d260a4258d18b3762adf90f7d3ffb9b0a707052374e4f735b4fdfc9a24ac4674"} Jul 29 04:00:15.733312 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:15.733016 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:15.754655 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:15.754610 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" podStartSLOduration=2.337270673 podStartE2EDuration="6.754594414s" totalImagesPullingTime="3.416354672s" totalInitContainerRuntime="1.000969069s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:15.75199988 +0000 UTC m=+298.044096242" watchObservedRunningTime="2026-07-29 04:00:15.754594414 +0000 UTC m=+298.046690776" Jul 29 04:00:18.074011 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:18.073980 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:00:18.079541 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:18.079515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:00:18.086177 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:18.086156 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 04:00:21.176032 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:21.175995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:22.544162 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:22.544127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:26.739750 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:26.739724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-fbcdv" Jul 29 04:00:28.760436 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:28.760396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:28.896292 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:28.896259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:29.439210 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.439170 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h9sc8"] Jul 29 04:00:29.442604 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.442582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:29.446427 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.446320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 04:00:29.447721 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.447699 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kxptb\"" Jul 29 04:00:29.454240 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.454218 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h9sc8"] Jul 29 04:00:29.481519 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.481494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:29.529517 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.529491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjd9\" (UniqueName: \"kubernetes.io/projected/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-kube-api-access-7vjd9\") pod \"odh-model-controller-7998c67dc4-h9sc8\" (UID: \"f981cf48-7ab5-4596-80a4-cd5910b8f5c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:29.529658 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.529551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-cert\") pod \"odh-model-controller-7998c67dc4-h9sc8\" (UID: \"f981cf48-7ab5-4596-80a4-cd5910b8f5c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:29.553741 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.553710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:29.630543 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.630510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjd9\" (UniqueName: \"kubernetes.io/projected/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-kube-api-access-7vjd9\") pod \"odh-model-controller-7998c67dc4-h9sc8\" (UID: \"f981cf48-7ab5-4596-80a4-cd5910b8f5c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:29.630714 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.630560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-cert\") pod \"odh-model-controller-7998c67dc4-h9sc8\" (UID: \"f981cf48-7ab5-4596-80a4-cd5910b8f5c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:29.630714 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:00:29.630675 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 04:00:29.630799 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:00:29.630735 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-cert podName:f981cf48-7ab5-4596-80a4-cd5910b8f5c1 nodeName:}" failed. No retries permitted until 2026-07-29 04:00:30.130716179 +0000 UTC m=+312.422812522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-cert") pod "odh-model-controller-7998c67dc4-h9sc8" (UID: "f981cf48-7ab5-4596-80a4-cd5910b8f5c1") : secret "odh-model-controller-webhook-cert" not found Jul 29 04:00:29.639523 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.639495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjd9\" (UniqueName: \"kubernetes.io/projected/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-kube-api-access-7vjd9\") pod \"odh-model-controller-7998c67dc4-h9sc8\" (UID: \"f981cf48-7ab5-4596-80a4-cd5910b8f5c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:29.872978 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.872945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:29.936271 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.936239 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8ksb"] Jul 29 04:00:29.939494 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.939478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:29.941826 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.941804 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 04:00:29.941962 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.941856 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5n5r9\"" Jul 29 04:00:29.949769 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:29.949750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8ksb"] Jul 29 04:00:30.033844 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.033794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad01a230-dc80-4b63-946c-df141ae371f6-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8ksb\" (UID: \"ad01a230-dc80-4b63-946c-df141ae371f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:30.034017 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.033950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5zh\" (UniqueName: \"kubernetes.io/projected/ad01a230-dc80-4b63-946c-df141ae371f6-kube-api-access-zs5zh\") pod \"model-serving-api-54d89bd5fc-v8ksb\" (UID: \"ad01a230-dc80-4b63-946c-df141ae371f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:30.053456 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.053428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:30.134596 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.134516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-cert\") pod \"odh-model-controller-7998c67dc4-h9sc8\" (UID: \"f981cf48-7ab5-4596-80a4-cd5910b8f5c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:30.134596 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.134556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5zh\" (UniqueName: \"kubernetes.io/projected/ad01a230-dc80-4b63-946c-df141ae371f6-kube-api-access-zs5zh\") pod \"model-serving-api-54d89bd5fc-v8ksb\" (UID: \"ad01a230-dc80-4b63-946c-df141ae371f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:30.134797 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.134632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad01a230-dc80-4b63-946c-df141ae371f6-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8ksb\" (UID: \"ad01a230-dc80-4b63-946c-df141ae371f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:30.137270 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.137246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad01a230-dc80-4b63-946c-df141ae371f6-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8ksb\" (UID: \"ad01a230-dc80-4b63-946c-df141ae371f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:30.137379 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.137317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f981cf48-7ab5-4596-80a4-cd5910b8f5c1-cert\") pod \"odh-model-controller-7998c67dc4-h9sc8\" (UID: \"f981cf48-7ab5-4596-80a4-cd5910b8f5c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:30.147968 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.147946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5zh\" (UniqueName: \"kubernetes.io/projected/ad01a230-dc80-4b63-946c-df141ae371f6-kube-api-access-zs5zh\") pod \"model-serving-api-54d89bd5fc-v8ksb\" (UID: \"ad01a230-dc80-4b63-946c-df141ae371f6\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:30.155237 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.155218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:30.249609 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.249580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:30.354982 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.354946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:30.385084 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.385061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8ksb"] Jul 29 04:00:30.388039 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:00:30.388005 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad01a230_dc80_4b63_946c_df141ae371f6.slice/crio-45246ca533f20e06c2a35e39d7d931ec3948fdb58ea7ab03e159feea16683815 WatchSource:0}: Error finding container 45246ca533f20e06c2a35e39d7d931ec3948fdb58ea7ab03e159feea16683815: Status 404 returned error can't find the container with id 45246ca533f20e06c2a35e39d7d931ec3948fdb58ea7ab03e159feea16683815 Jul 29 04:00:30.389901 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.389864 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:00:30.493694 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.493637 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h9sc8"] Jul 29 04:00:30.496381 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:00:30.496329 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf981cf48_7ab5_4596_80a4_cd5910b8f5c1.slice/crio-50a120d9883bcacd635d27a29f0753641717233e80094dc9ecafdfe5f27eef84 WatchSource:0}: Error finding container 50a120d9883bcacd635d27a29f0753641717233e80094dc9ecafdfe5f27eef84: Status 404 returned error can't find the container with id 50a120d9883bcacd635d27a29f0753641717233e80094dc9ecafdfe5f27eef84 Jul 29 04:00:30.802503 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.802426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" event={"ID":"ad01a230-dc80-4b63-946c-df141ae371f6","Type":"ContainerStarted","Data":"45246ca533f20e06c2a35e39d7d931ec3948fdb58ea7ab03e159feea16683815"} Jul 29 04:00:30.803654 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:30.803625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" event={"ID":"f981cf48-7ab5-4596-80a4-cd5910b8f5c1","Type":"ContainerStarted","Data":"50a120d9883bcacd635d27a29f0753641717233e80094dc9ecafdfe5f27eef84"} Jul 29 04:00:33.241572 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:33.241542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:34.828757 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:34.828724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" event={"ID":"f981cf48-7ab5-4596-80a4-cd5910b8f5c1","Type":"ContainerStarted","Data":"556d3e80a6ac3c142bb38abea29ee215429647f3958d7fb9e71d12814e834c35"} Jul 29 04:00:34.829248 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:34.828906 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:34.830439 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:34.830412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" event={"ID":"ad01a230-dc80-4b63-946c-df141ae371f6","Type":"ContainerStarted","Data":"561c31ffe5e32cee22b6dd9e73f1a729fd849a1be5a577fa4b128f7dbdb5d054"} Jul 29 04:00:34.830554 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:34.830539 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:34.844369 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:34.844331 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" podStartSLOduration=2.4759074549999998 podStartE2EDuration="5.844318628s" totalImagesPullingTime="3.368411173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:34.843296657 +0000 UTC m=+317.135393020" watchObservedRunningTime="2026-07-29 04:00:34.844318628 +0000 UTC m=+317.136414982" Jul 29 04:00:34.858645 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:34.858602 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" podStartSLOduration=2.4312848000000002 podStartE2EDuration="5.858590217s" totalImagesPullingTime="3.427305417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:34.857553587 +0000 UTC m=+317.149649950" watchObservedRunningTime="2026-07-29 04:00:34.858590217 +0000 UTC m=+317.150686578" Jul 29 04:00:39.499799 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.499763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:39.547792 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.547759 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5"] Jul 29 04:00:39.550047 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.550030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.552448 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.552431 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 29 04:00:39.552543 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.552476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-c4nsr\"" Jul 29 04:00:39.557181 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.557157 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg"] Jul 29 04:00:39.559581 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.559563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:39.561781 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.561764 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 29 04:00:39.562971 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.562923 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9zfdx\"" Jul 29 04:00:39.564104 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.564084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5"] Jul 29 04:00:39.573003 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.572984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg"] Jul 29 04:00:39.612863 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.612838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:39.615006 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.614985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrws\" (UniqueName: \"kubernetes.io/projected/ccf78918-f12e-43bc-b98d-8c09acab26ae-kube-api-access-pwrws\") pod \"kserve-controller-manager-d6f4b65b8-5hjg5\" (UID: \"ccf78918-f12e-43bc-b98d-8c09acab26ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.615112 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.615034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccf78918-f12e-43bc-b98d-8c09acab26ae-cert\") pod \"kserve-controller-manager-d6f4b65b8-5hjg5\" (UID: \"ccf78918-f12e-43bc-b98d-8c09acab26ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.716465 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.716431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58fdab89-2cb1-4b21-a69e-4510fa300b04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q9xfg\" (UID: \"58fdab89-2cb1-4b21-a69e-4510fa300b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:39.716465 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.716469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrws\" (UniqueName: \"kubernetes.io/projected/ccf78918-f12e-43bc-b98d-8c09acab26ae-kube-api-access-pwrws\") pod \"kserve-controller-manager-d6f4b65b8-5hjg5\" (UID: \"ccf78918-f12e-43bc-b98d-8c09acab26ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.716703 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.716543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccf78918-f12e-43bc-b98d-8c09acab26ae-cert\") pod \"kserve-controller-manager-d6f4b65b8-5hjg5\" (UID: \"ccf78918-f12e-43bc-b98d-8c09acab26ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.716761 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.716719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6fc\" (UniqueName: \"kubernetes.io/projected/58fdab89-2cb1-4b21-a69e-4510fa300b04-kube-api-access-wr6fc\") pod \"llmisvc-controller-manager-777c6bd7d5-q9xfg\" (UID: \"58fdab89-2cb1-4b21-a69e-4510fa300b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:39.719107 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.719079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccf78918-f12e-43bc-b98d-8c09acab26ae-cert\") pod \"kserve-controller-manager-d6f4b65b8-5hjg5\" (UID: \"ccf78918-f12e-43bc-b98d-8c09acab26ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.731775 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.731752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrws\" (UniqueName: \"kubernetes.io/projected/ccf78918-f12e-43bc-b98d-8c09acab26ae-kube-api-access-pwrws\") pod \"kserve-controller-manager-d6f4b65b8-5hjg5\" (UID: \"ccf78918-f12e-43bc-b98d-8c09acab26ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.817697 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.817621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58fdab89-2cb1-4b21-a69e-4510fa300b04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q9xfg\" (UID: \"58fdab89-2cb1-4b21-a69e-4510fa300b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:39.817830 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:00:39.817755 2576 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 29 04:00:39.817830 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.817755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6fc\" (UniqueName: \"kubernetes.io/projected/58fdab89-2cb1-4b21-a69e-4510fa300b04-kube-api-access-wr6fc\") pod \"llmisvc-controller-manager-777c6bd7d5-q9xfg\" (UID: \"58fdab89-2cb1-4b21-a69e-4510fa300b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:39.817830 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:00:39.817821 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58fdab89-2cb1-4b21-a69e-4510fa300b04-cert podName:58fdab89-2cb1-4b21-a69e-4510fa300b04 nodeName:}" failed. No retries permitted until 2026-07-29 04:00:40.317800591 +0000 UTC m=+322.609896936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/58fdab89-2cb1-4b21-a69e-4510fa300b04-cert") pod "llmisvc-controller-manager-777c6bd7d5-q9xfg" (UID: "58fdab89-2cb1-4b21-a69e-4510fa300b04") : secret "llmisvc-webhook-server-cert" not found Jul 29 04:00:39.827732 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.827702 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt"] Jul 29 04:00:39.830351 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.830333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:39.833968 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.833950 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-btzts\"" Jul 29 04:00:39.834780 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.834758 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 04:00:39.835112 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.835096 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 04:00:39.836385 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.836364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6fc\" (UniqueName: \"kubernetes.io/projected/58fdab89-2cb1-4b21-a69e-4510fa300b04-kube-api-access-wr6fc\") pod \"llmisvc-controller-manager-777c6bd7d5-q9xfg\" (UID: \"58fdab89-2cb1-4b21-a69e-4510fa300b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:39.854150 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.854129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt"] Jul 29 04:00:39.860977 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.860960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:39.920328 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.920038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqg9\" (UniqueName: \"kubernetes.io/projected/73085c0c-0c12-49e1-b064-17b725ed001e-kube-api-access-hmqg9\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:39.920328 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.920127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73085c0c-0c12-49e1-b064-17b725ed001e-tmp\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:39.920328 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.920245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73085c0c-0c12-49e1-b064-17b725ed001e-tls-certs\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:39.991576 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:39.991555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5"] Jul 29 04:00:39.993772 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:00:39.993744 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf78918_f12e_43bc_b98d_8c09acab26ae.slice/crio-84a176e5912d2be44dbc05678565de2eac6ea1c6400f32b35212f9d9136950b6 WatchSource:0}: Error finding container 84a176e5912d2be44dbc05678565de2eac6ea1c6400f32b35212f9d9136950b6: Status 404 returned error can't find the container with id 84a176e5912d2be44dbc05678565de2eac6ea1c6400f32b35212f9d9136950b6 Jul 29 04:00:40.020754 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.020736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73085c0c-0c12-49e1-b064-17b725ed001e-tls-certs\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:40.020870 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.020767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqg9\" (UniqueName: \"kubernetes.io/projected/73085c0c-0c12-49e1-b064-17b725ed001e-kube-api-access-hmqg9\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:40.020870 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.020799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73085c0c-0c12-49e1-b064-17b725ed001e-tmp\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:40.023081 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.023057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73085c0c-0c12-49e1-b064-17b725ed001e-tmp\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:40.023207 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.023191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73085c0c-0c12-49e1-b064-17b725ed001e-tls-certs\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:40.029758 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.029740 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqg9\" (UniqueName: \"kubernetes.io/projected/73085c0c-0c12-49e1-b064-17b725ed001e-kube-api-access-hmqg9\") pod \"kube-auth-proxy-664ccd966c-8r9lt\" (UID: \"73085c0c-0c12-49e1-b064-17b725ed001e\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:40.147947 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.147925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" Jul 29 04:00:40.187793 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.187728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:40.243505 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.243468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:40.284437 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:00:40.284407 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73085c0c_0c12_49e1_b064_17b725ed001e.slice/crio-1ad4d52778f2d60f28340c26eb8223805a7a485263c4f2d68e0dcdb59ec0c429 WatchSource:0}: Error finding container 1ad4d52778f2d60f28340c26eb8223805a7a485263c4f2d68e0dcdb59ec0c429: Status 404 returned error can't find the container with id 1ad4d52778f2d60f28340c26eb8223805a7a485263c4f2d68e0dcdb59ec0c429 Jul 29 04:00:40.284801 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.284779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt"] Jul 29 04:00:40.324236 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.324211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58fdab89-2cb1-4b21-a69e-4510fa300b04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q9xfg\" (UID: \"58fdab89-2cb1-4b21-a69e-4510fa300b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:40.325348 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.325326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:40.326740 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.326723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58fdab89-2cb1-4b21-a69e-4510fa300b04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q9xfg\" (UID: \"58fdab89-2cb1-4b21-a69e-4510fa300b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:40.470726 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.470656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:40.613835 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.613807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg"] Jul 29 04:00:40.616236 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:00:40.616200 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58fdab89_2cb1_4b21_a69e_4510fa300b04.slice/crio-e97bde37adb809fd85b7e1709867f6ece8c069b8646ef8df7768b4d71608d875 WatchSource:0}: Error finding container e97bde37adb809fd85b7e1709867f6ece8c069b8646ef8df7768b4d71608d875: Status 404 returned error can't find the container with id e97bde37adb809fd85b7e1709867f6ece8c069b8646ef8df7768b4d71608d875 Jul 29 04:00:40.755915 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.755812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:40.862119 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.862087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" event={"ID":"73085c0c-0c12-49e1-b064-17b725ed001e","Type":"ContainerStarted","Data":"1ad4d52778f2d60f28340c26eb8223805a7a485263c4f2d68e0dcdb59ec0c429"} Jul 29 04:00:40.863199 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.863175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" event={"ID":"58fdab89-2cb1-4b21-a69e-4510fa300b04","Type":"ContainerStarted","Data":"e97bde37adb809fd85b7e1709867f6ece8c069b8646ef8df7768b4d71608d875"} Jul 29 04:00:40.864300 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.864280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" event={"ID":"ccf78918-f12e-43bc-b98d-8c09acab26ae","Type":"ContainerStarted","Data":"84a176e5912d2be44dbc05678565de2eac6ea1c6400f32b35212f9d9136950b6"} Jul 29 04:00:40.866361 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:40.866335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:41.406233 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.406177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:41.788492 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.788381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:41.815543 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.815507 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt"] Jul 29 04:00:41.819592 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.819567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:41.825133 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.825101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zp7dl\"" Jul 29 04:00:41.826097 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.825926 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 04:00:41.834528 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.834347 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 04:00:41.837072 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.837048 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 04:00:41.840371 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.840346 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 04:00:41.860432 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.860342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt"] Jul 29 04:00:41.903061 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.902869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:41.942931 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.942863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:41.943076 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.942960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdt4\" (UniqueName: \"kubernetes.io/projected/763a30c4-5558-486d-9563-e6def2625bfa-kube-api-access-prdt4\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:41.943076 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.943018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/763a30c4-5558-486d-9563-e6def2625bfa-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:41.943076 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.943068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763a30c4-5558-486d-9563-e6def2625bfa-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:41.943196 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.943107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:41.943196 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.943152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:41.943196 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:41.943185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/763a30c4-5558-486d-9563-e6def2625bfa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.044644 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.044566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.044644 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.044624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdt4\" (UniqueName: \"kubernetes.io/projected/763a30c4-5558-486d-9563-e6def2625bfa-kube-api-access-prdt4\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.044838 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.044683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/763a30c4-5558-486d-9563-e6def2625bfa-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.044838 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.044737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763a30c4-5558-486d-9563-e6def2625bfa-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.044838 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.044782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.044838 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.044830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.045075 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.044861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/763a30c4-5558-486d-9563-e6def2625bfa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.045640 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.045615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/763a30c4-5558-486d-9563-e6def2625bfa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.050499 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.050478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.052912 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.052872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.053369 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.053347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/763a30c4-5558-486d-9563-e6def2625bfa-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.053679 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.053655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/763a30c4-5558-486d-9563-e6def2625bfa-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.057906 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.057866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763a30c4-5558-486d-9563-e6def2625bfa-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.058030 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.058004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdt4\" (UniqueName: \"kubernetes.io/projected/763a30c4-5558-486d-9563-e6def2625bfa-kube-api-access-prdt4\") pod \"istiod-openshift-gateway-5677ccb49b-b69pt\" (UID: \"763a30c4-5558-486d-9563-e6def2625bfa\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.134038 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.134008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:42.313576 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:42.313535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt"] Jul 29 04:00:42.950770 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:00:42.950735 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763a30c4_5558_486d_9563_e6def2625bfa.slice/crio-a04fca406d2284fd562195b89690d42786711565f7c6bdf38124637b3a53cc8f WatchSource:0}: Error finding container a04fca406d2284fd562195b89690d42786711565f7c6bdf38124637b3a53cc8f: Status 404 returned error can't find the container with id a04fca406d2284fd562195b89690d42786711565f7c6bdf38124637b3a53cc8f Jul 29 04:00:43.048178 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:43.048122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:43.884214 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:43.884173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" event={"ID":"763a30c4-5558-486d-9563-e6def2625bfa","Type":"ContainerStarted","Data":"a04fca406d2284fd562195b89690d42786711565f7c6bdf38124637b3a53cc8f"} Jul 29 04:00:44.893465 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.893415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" event={"ID":"73085c0c-0c12-49e1-b064-17b725ed001e","Type":"ContainerStarted","Data":"aca0a076ca06e8cd955bcc06c6f61e2f22c0f6fe837c3e94e0e4e5bd2afbe499"} Jul 29 04:00:44.896093 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.896007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" event={"ID":"58fdab89-2cb1-4b21-a69e-4510fa300b04","Type":"ContainerStarted","Data":"7afcaa83d5dd8ab2ee64f145f23550b7e4e7ef5ca4cbaddb71327b3fd7b5a17d"} Jul 29 04:00:44.896250 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.896129 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:00:44.898154 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.898124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" event={"ID":"ccf78918-f12e-43bc-b98d-8c09acab26ae","Type":"ContainerStarted","Data":"a7a0dd1d190fc208efd62d49ad9aab293740d2fd43a2705db305e13c7630f75b"} Jul 29 04:00:44.898407 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.898356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:00:44.910326 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.910284 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-664ccd966c-8r9lt" podStartSLOduration=2.061959211 podStartE2EDuration="5.910260013s" totalImagesPullingTime="3.848300802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:44.908620607 +0000 UTC m=+327.200716988" watchObservedRunningTime="2026-07-29 04:00:44.910260013 +0000 UTC m=+327.202356385" Jul 29 04:00:44.928637 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.928596 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" podStartSLOduration=2.400015792 podStartE2EDuration="5.928579926s" totalImagesPullingTime="3.528564134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:44.928443346 +0000 UTC m=+327.220539715" watchObservedRunningTime="2026-07-29 04:00:44.928579926 +0000 UTC m=+327.220676287" Jul 29 04:00:44.947035 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:44.946945 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" podStartSLOduration=1.806207372 podStartE2EDuration="5.946925825s" totalImagesPullingTime="4.140718453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:44.944855754 +0000 UTC m=+327.236952123" watchObservedRunningTime="2026-07-29 04:00:44.946925825 +0000 UTC m=+327.239022194" Jul 29 04:00:45.838000 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:45.837969 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-h9sc8" Jul 29 04:00:45.839909 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:45.839796 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-v8ksb" Jul 29 04:00:46.035050 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:46.035013 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:00:46.035390 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:46.035083 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:00:46.585799 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:46.585706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:46.646375 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:46.646330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:46.918966 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:46.918931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" event={"ID":"763a30c4-5558-486d-9563-e6def2625bfa","Type":"ContainerStarted","Data":"ddf10f9e64401f2783d8e6da869779a4314fef408eb51c1d5fe344d8e2f3d5bd"} Jul 29 04:00:46.919169 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:46.919069 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:00:46.944144 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:46.944102 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" podStartSLOduration=2.862368656 podStartE2EDuration="5.944077481s" totalImagesPullingTime="3.081708825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:46.94318961 +0000 UTC m=+329.235285976" watchObservedRunningTime="2026-07-29 04:00:46.944077481 +0000 UTC m=+329.236173842" Jul 29 04:00:47.092802 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:47.092759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:47.187971 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:47.187870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:47.289992 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:47.289950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:47.417083 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:47.417033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:47.516600 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:47.516532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:00:47.925769 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:00:47.925736 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b69pt" Jul 29 04:01:15.914378 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:15.914307 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q9xfg" Jul 29 04:01:15.917548 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:15.917530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5hjg5" Jul 29 04:01:30.247559 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:30.247522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:01:30.343616 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:30.343580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:01:30.442437 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:30.442397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:01:45.319283 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.319248 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf"] Jul 29 04:01:45.321865 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.321850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.324420 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.324395 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 04:01:45.324530 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.324512 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 04:01:45.325650 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.325634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jbt2t\"" Jul 29 04:01:45.330023 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.330004 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf"] Jul 29 04:01:45.482775 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.482737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.482775 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.482777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5pz\" (UniqueName: \"kubernetes.io/projected/3a214469-3308-42d4-b967-9d9b36e96b3b-kube-api-access-pm5pz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.483005 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.482874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.583333 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.583260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5pz\" (UniqueName: \"kubernetes.io/projected/3a214469-3308-42d4-b967-9d9b36e96b3b-kube-api-access-pm5pz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.583462 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.583340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.583462 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.583408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.583704 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.583685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.583736 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.583715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.591321 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.591299 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5pz\" (UniqueName: \"kubernetes.io/projected/3a214469-3308-42d4-b967-9d9b36e96b3b-kube-api-access-pm5pz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.632119 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.632077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:45.714034 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.714006 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v"] Jul 29 04:01:45.716711 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.716692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.728201 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.728111 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v"] Jul 29 04:01:45.772462 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.772435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf"] Jul 29 04:01:45.774356 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:01:45.774330 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a214469_3308_42d4_b967_9d9b36e96b3b.slice/crio-91c67c4bd61c64528172d3f17a7919817748a6035efcf27fd130e16a1bcc11f9 WatchSource:0}: Error finding container 91c67c4bd61c64528172d3f17a7919817748a6035efcf27fd130e16a1bcc11f9: Status 404 returned error can't find the container with id 91c67c4bd61c64528172d3f17a7919817748a6035efcf27fd130e16a1bcc11f9 Jul 29 04:01:45.886310 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.886284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.886420 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.886325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5k5x\" (UniqueName: \"kubernetes.io/projected/05790ad2-5d86-4ebd-b90c-83c79f96ff07-kube-api-access-k5k5x\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.886420 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.886356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.987152 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.987117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.987152 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.987158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5k5x\" (UniqueName: \"kubernetes.io/projected/05790ad2-5d86-4ebd-b90c-83c79f96ff07-kube-api-access-k5k5x\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.987394 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.987283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.987494 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.987477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.987586 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.987571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:45.995039 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:45.995019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5k5x\" (UniqueName: \"kubernetes.io/projected/05790ad2-5d86-4ebd-b90c-83c79f96ff07-kube-api-access-k5k5x\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:46.029858 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.029832 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:46.159586 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.159560 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v"] Jul 29 04:01:46.161793 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:01:46.161767 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05790ad2_5d86_4ebd_b90c_83c79f96ff07.slice/crio-453f457be52258e172850425bf18eb8bb40cb5bec112c88b4f5b90cc552c9d24 WatchSource:0}: Error finding container 453f457be52258e172850425bf18eb8bb40cb5bec112c88b4f5b90cc552c9d24: Status 404 returned error can't find the container with id 453f457be52258e172850425bf18eb8bb40cb5bec112c88b4f5b90cc552c9d24 Jul 29 04:01:46.203722 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.203696 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerID="b647b22f383bb33053995b23da6047448a9c24df47e4efe2bd3c0bf126502836" exitCode=0 Jul 29 04:01:46.203825 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.203772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" event={"ID":"3a214469-3308-42d4-b967-9d9b36e96b3b","Type":"ContainerDied","Data":"b647b22f383bb33053995b23da6047448a9c24df47e4efe2bd3c0bf126502836"} Jul 29 04:01:46.203825 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.203814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" event={"ID":"3a214469-3308-42d4-b967-9d9b36e96b3b","Type":"ContainerStarted","Data":"91c67c4bd61c64528172d3f17a7919817748a6035efcf27fd130e16a1bcc11f9"} Jul 29 04:01:46.205121 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.205103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" event={"ID":"05790ad2-5d86-4ebd-b90c-83c79f96ff07","Type":"ContainerStarted","Data":"453f457be52258e172850425bf18eb8bb40cb5bec112c88b4f5b90cc552c9d24"} Jul 29 04:01:46.311205 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.311178 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw"] Jul 29 04:01:46.313603 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.313584 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.322590 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.322567 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw"] Jul 29 04:01:46.492601 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.492572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8flz\" (UniqueName: \"kubernetes.io/projected/399cdbd9-8aa0-4f0d-8b29-191573cc1699-kube-api-access-z8flz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.492752 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.492620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.492752 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.492702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.593607 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.593582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.593778 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.593622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8flz\" (UniqueName: \"kubernetes.io/projected/399cdbd9-8aa0-4f0d-8b29-191573cc1699-kube-api-access-z8flz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.593778 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.593661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.594066 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.594045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.594142 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.594078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.601587 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.601567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8flz\" (UniqueName: \"kubernetes.io/projected/399cdbd9-8aa0-4f0d-8b29-191573cc1699-kube-api-access-z8flz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.638869 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.638844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:46.768281 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.768259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw"] Jul 29 04:01:46.770611 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:01:46.770575 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399cdbd9_8aa0_4f0d_8b29_191573cc1699.slice/crio-320d42309668ec3daae98d2264acdc597752addb32a07bf923364c57b80c36e6 WatchSource:0}: Error finding container 320d42309668ec3daae98d2264acdc597752addb32a07bf923364c57b80c36e6: Status 404 returned error can't find the container with id 320d42309668ec3daae98d2264acdc597752addb32a07bf923364c57b80c36e6 Jul 29 04:01:46.919149 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.919121 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z"] Jul 29 04:01:46.921667 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.921650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:46.931731 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:46.931709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z"] Jul 29 04:01:47.099133 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.099063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8tv\" (UniqueName: \"kubernetes.io/projected/257ae2f6-4912-4362-b500-b886e8a3c5be-kube-api-access-xd8tv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.099133 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.099112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.099311 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.099226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.199832 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.199807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.199955 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.199863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8tv\" (UniqueName: \"kubernetes.io/projected/257ae2f6-4912-4362-b500-b886e8a3c5be-kube-api-access-xd8tv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.199955 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.199940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.200216 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.200197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.200303 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.200253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.209414 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.209393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8tv\" (UniqueName: \"kubernetes.io/projected/257ae2f6-4912-4362-b500-b886e8a3c5be-kube-api-access-xd8tv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.211685 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.211663 2576 generic.go:356] "Generic (PLEG): container finished" podID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerID="ec511d2502d6680b068477aad18e5de133205db19b1133125d8e7e1d1e6a538e" exitCode=0 Jul 29 04:01:47.211780 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.211745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" event={"ID":"399cdbd9-8aa0-4f0d-8b29-191573cc1699","Type":"ContainerDied","Data":"ec511d2502d6680b068477aad18e5de133205db19b1133125d8e7e1d1e6a538e"} Jul 29 04:01:47.211851 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.211823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" event={"ID":"399cdbd9-8aa0-4f0d-8b29-191573cc1699","Type":"ContainerStarted","Data":"320d42309668ec3daae98d2264acdc597752addb32a07bf923364c57b80c36e6"} Jul 29 04:01:47.213415 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.213393 2576 generic.go:356] "Generic (PLEG): container finished" podID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerID="abc663c34c3b1bb267e2dffc5ba11d2e1a9d4ca100b4db7056141470fe3c188d" exitCode=0 Jul 29 04:01:47.213497 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.213423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" event={"ID":"05790ad2-5d86-4ebd-b90c-83c79f96ff07","Type":"ContainerDied","Data":"abc663c34c3b1bb267e2dffc5ba11d2e1a9d4ca100b4db7056141470fe3c188d"} Jul 29 04:01:47.230725 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.230687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:47.362066 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:47.362041 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z"] Jul 29 04:01:47.363774 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:01:47.363749 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ae2f6_4912_4362_b500_b886e8a3c5be.slice/crio-07779d7606887f4940b00156204cd472875ff1c46772d7fd51c4ad3e89f0be61 WatchSource:0}: Error finding container 07779d7606887f4940b00156204cd472875ff1c46772d7fd51c4ad3e89f0be61: Status 404 returned error can't find the container with id 07779d7606887f4940b00156204cd472875ff1c46772d7fd51c4ad3e89f0be61 Jul 29 04:01:48.219515 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.219486 2576 generic.go:356] "Generic (PLEG): container finished" podID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerID="1799e1b3e3ef064985cdca857d3f7ef648f604079b3e5af2638d0ffdbbe243b2" exitCode=0 Jul 29 04:01:48.219627 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.219575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" event={"ID":"257ae2f6-4912-4362-b500-b886e8a3c5be","Type":"ContainerDied","Data":"1799e1b3e3ef064985cdca857d3f7ef648f604079b3e5af2638d0ffdbbe243b2"} Jul 29 04:01:48.219691 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.219632 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" event={"ID":"257ae2f6-4912-4362-b500-b886e8a3c5be","Type":"ContainerStarted","Data":"07779d7606887f4940b00156204cd472875ff1c46772d7fd51c4ad3e89f0be61"} Jul 29 04:01:48.221726 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.221702 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerID="3fe35a8f5595de8138d65136e839796264adbe390a850cdd314af159e82c1ffe" exitCode=0 Jul 29 04:01:48.221834 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.221785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" event={"ID":"3a214469-3308-42d4-b967-9d9b36e96b3b","Type":"ContainerDied","Data":"3fe35a8f5595de8138d65136e839796264adbe390a850cdd314af159e82c1ffe"} Jul 29 04:01:48.223907 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.223868 2576 generic.go:356] "Generic (PLEG): container finished" podID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerID="6f6e800010aa91385046f7dee861ee675d0ba1edd878972b5834a1295c5cdcff" exitCode=0 Jul 29 04:01:48.224019 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.223923 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" event={"ID":"05790ad2-5d86-4ebd-b90c-83c79f96ff07","Type":"ContainerDied","Data":"6f6e800010aa91385046f7dee861ee675d0ba1edd878972b5834a1295c5cdcff"} Jul 29 04:01:48.226301 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.226276 2576 generic.go:356] "Generic (PLEG): container finished" podID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerID="ed29d5bdf4fc4994d7300955b45ffd3b4101b9a07d7429f9d218efe1661a50bb" exitCode=0 Jul 29 04:01:48.226394 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:48.226322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" event={"ID":"399cdbd9-8aa0-4f0d-8b29-191573cc1699","Type":"ContainerDied","Data":"ed29d5bdf4fc4994d7300955b45ffd3b4101b9a07d7429f9d218efe1661a50bb"} Jul 29 04:01:49.233517 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.233450 2576 generic.go:356] "Generic (PLEG): container finished" podID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerID="8cac9db857ff62224f2db22851a3cea5d66e3a987712678b948b858a95dd5770" exitCode=0 Jul 29 04:01:49.233985 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.233539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" event={"ID":"257ae2f6-4912-4362-b500-b886e8a3c5be","Type":"ContainerDied","Data":"8cac9db857ff62224f2db22851a3cea5d66e3a987712678b948b858a95dd5770"} Jul 29 04:01:49.235734 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.235712 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerID="876add2b75f9a7c42b5a0b33919d29d57be9bfdd0a6230c9c991adfc5559d492" exitCode=0 Jul 29 04:01:49.235821 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.235790 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" event={"ID":"3a214469-3308-42d4-b967-9d9b36e96b3b","Type":"ContainerDied","Data":"876add2b75f9a7c42b5a0b33919d29d57be9bfdd0a6230c9c991adfc5559d492"} Jul 29 04:01:49.237915 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.237875 2576 generic.go:356] "Generic (PLEG): container finished" podID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerID="1536f0da7be8130de70a9dda5a6751b5ab46241fe66c4e368fd953011d2654db" exitCode=0 Jul 29 04:01:49.238015 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.237987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" event={"ID":"05790ad2-5d86-4ebd-b90c-83c79f96ff07","Type":"ContainerDied","Data":"1536f0da7be8130de70a9dda5a6751b5ab46241fe66c4e368fd953011d2654db"} Jul 29 04:01:49.239943 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.239924 2576 generic.go:356] "Generic (PLEG): container finished" podID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerID="6362bd22b3e12f46f9ec737d38dcd229f0986442af0ae0e09ddb73cfdd27bb6b" exitCode=0 Jul 29 04:01:49.240020 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.239952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" event={"ID":"399cdbd9-8aa0-4f0d-8b29-191573cc1699","Type":"ContainerDied","Data":"6362bd22b3e12f46f9ec737d38dcd229f0986442af0ae0e09ddb73cfdd27bb6b"} Jul 29 04:01:49.399343 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.399314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:01:49.450867 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:49.450839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:01:50.248632 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.248603 2576 generic.go:356] "Generic (PLEG): container finished" podID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerID="c3e1a3aca8ed5c16c96dd9536d9e4cd544d1c984e9e2934825cabd88bde3c20a" exitCode=0 Jul 29 04:01:50.249062 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.248696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" event={"ID":"257ae2f6-4912-4362-b500-b886e8a3c5be","Type":"ContainerDied","Data":"c3e1a3aca8ed5c16c96dd9536d9e4cd544d1c984e9e2934825cabd88bde3c20a"} Jul 29 04:01:50.399691 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.399666 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:50.444096 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.444070 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:50.446758 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.446741 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:50.528279 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.528201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-util\") pod \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " Jul 29 04:01:50.528279 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.528231 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-bundle\") pod \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " Jul 29 04:01:50.528279 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.528249 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5k5x\" (UniqueName: \"kubernetes.io/projected/05790ad2-5d86-4ebd-b90c-83c79f96ff07-kube-api-access-k5k5x\") pod \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\" (UID: \"05790ad2-5d86-4ebd-b90c-83c79f96ff07\") " Jul 29 04:01:50.528652 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.528615 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-bundle" (OuterVolumeSpecName: "bundle") pod "05790ad2-5d86-4ebd-b90c-83c79f96ff07" (UID: "05790ad2-5d86-4ebd-b90c-83c79f96ff07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:50.530565 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.530544 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05790ad2-5d86-4ebd-b90c-83c79f96ff07-kube-api-access-k5k5x" (OuterVolumeSpecName: "kube-api-access-k5k5x") pod "05790ad2-5d86-4ebd-b90c-83c79f96ff07" (UID: "05790ad2-5d86-4ebd-b90c-83c79f96ff07"). InnerVolumeSpecName "kube-api-access-k5k5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:01:50.533441 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.533418 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-util" (OuterVolumeSpecName: "util") pod "05790ad2-5d86-4ebd-b90c-83c79f96ff07" (UID: "05790ad2-5d86-4ebd-b90c-83c79f96ff07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:50.629594 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-bundle\") pod \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " Jul 29 04:01:50.629594 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629592 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-util\") pod \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " Jul 29 04:01:50.629937 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629646 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-util\") pod \"3a214469-3308-42d4-b967-9d9b36e96b3b\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " Jul 29 04:01:50.629937 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629683 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm5pz\" (UniqueName: \"kubernetes.io/projected/3a214469-3308-42d4-b967-9d9b36e96b3b-kube-api-access-pm5pz\") pod \"3a214469-3308-42d4-b967-9d9b36e96b3b\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " Jul 29 04:01:50.629937 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629710 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8flz\" (UniqueName: \"kubernetes.io/projected/399cdbd9-8aa0-4f0d-8b29-191573cc1699-kube-api-access-z8flz\") pod \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\" (UID: \"399cdbd9-8aa0-4f0d-8b29-191573cc1699\") " Jul 29 04:01:50.629937 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-bundle\") pod \"3a214469-3308-42d4-b967-9d9b36e96b3b\" (UID: \"3a214469-3308-42d4-b967-9d9b36e96b3b\") " Jul 29 04:01:50.630152 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629966 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-util\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.630152 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.629987 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05790ad2-5d86-4ebd-b90c-83c79f96ff07-bundle\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.630152 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.630000 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5k5x\" (UniqueName: \"kubernetes.io/projected/05790ad2-5d86-4ebd-b90c-83c79f96ff07-kube-api-access-k5k5x\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.630365 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.630284 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-bundle" (OuterVolumeSpecName: "bundle") pod "399cdbd9-8aa0-4f0d-8b29-191573cc1699" (UID: "399cdbd9-8aa0-4f0d-8b29-191573cc1699"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:50.630490 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.630471 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-bundle" (OuterVolumeSpecName: "bundle") pod "3a214469-3308-42d4-b967-9d9b36e96b3b" (UID: "3a214469-3308-42d4-b967-9d9b36e96b3b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:50.632589 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.632562 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399cdbd9-8aa0-4f0d-8b29-191573cc1699-kube-api-access-z8flz" (OuterVolumeSpecName: "kube-api-access-z8flz") pod "399cdbd9-8aa0-4f0d-8b29-191573cc1699" (UID: "399cdbd9-8aa0-4f0d-8b29-191573cc1699"). InnerVolumeSpecName "kube-api-access-z8flz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:01:50.632699 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.632596 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a214469-3308-42d4-b967-9d9b36e96b3b-kube-api-access-pm5pz" (OuterVolumeSpecName: "kube-api-access-pm5pz") pod "3a214469-3308-42d4-b967-9d9b36e96b3b" (UID: "3a214469-3308-42d4-b967-9d9b36e96b3b"). InnerVolumeSpecName "kube-api-access-pm5pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:01:50.638833 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.638807 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-util" (OuterVolumeSpecName: "util") pod "399cdbd9-8aa0-4f0d-8b29-191573cc1699" (UID: "399cdbd9-8aa0-4f0d-8b29-191573cc1699"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:50.639497 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.639479 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-util" (OuterVolumeSpecName: "util") pod "3a214469-3308-42d4-b967-9d9b36e96b3b" (UID: "3a214469-3308-42d4-b967-9d9b36e96b3b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:50.731175 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.731141 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-bundle\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.731175 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.731172 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/399cdbd9-8aa0-4f0d-8b29-191573cc1699-util\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.731175 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.731180 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-util\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.731175 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.731188 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5pz\" (UniqueName: \"kubernetes.io/projected/3a214469-3308-42d4-b967-9d9b36e96b3b-kube-api-access-pm5pz\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.731420 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.731197 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8flz\" (UniqueName: \"kubernetes.io/projected/399cdbd9-8aa0-4f0d-8b29-191573cc1699-kube-api-access-z8flz\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:50.731420 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:50.731205 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a214469-3308-42d4-b967-9d9b36e96b3b-bundle\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:51.255958 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.255925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" event={"ID":"3a214469-3308-42d4-b967-9d9b36e96b3b","Type":"ContainerDied","Data":"91c67c4bd61c64528172d3f17a7919817748a6035efcf27fd130e16a1bcc11f9"} Jul 29 04:01:51.256427 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.255970 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c67c4bd61c64528172d3f17a7919817748a6035efcf27fd130e16a1bcc11f9" Jul 29 04:01:51.256427 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.255933 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf" Jul 29 04:01:51.257931 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.257869 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" Jul 29 04:01:51.257931 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.257871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v" event={"ID":"05790ad2-5d86-4ebd-b90c-83c79f96ff07","Type":"ContainerDied","Data":"453f457be52258e172850425bf18eb8bb40cb5bec112c88b4f5b90cc552c9d24"} Jul 29 04:01:51.258115 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.257939 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453f457be52258e172850425bf18eb8bb40cb5bec112c88b4f5b90cc552c9d24" Jul 29 04:01:51.259876 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.259854 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" Jul 29 04:01:51.260007 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.259913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw" event={"ID":"399cdbd9-8aa0-4f0d-8b29-191573cc1699","Type":"ContainerDied","Data":"320d42309668ec3daae98d2264acdc597752addb32a07bf923364c57b80c36e6"} Jul 29 04:01:51.260007 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.259953 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="320d42309668ec3daae98d2264acdc597752addb32a07bf923364c57b80c36e6" Jul 29 04:01:51.321798 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.321768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:01:51.392956 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.392932 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:51.538856 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.538782 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd8tv\" (UniqueName: \"kubernetes.io/projected/257ae2f6-4912-4362-b500-b886e8a3c5be-kube-api-access-xd8tv\") pod \"257ae2f6-4912-4362-b500-b886e8a3c5be\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " Jul 29 04:01:51.538856 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.538838 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-util\") pod \"257ae2f6-4912-4362-b500-b886e8a3c5be\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " Jul 29 04:01:51.539090 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.538921 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-bundle\") pod \"257ae2f6-4912-4362-b500-b886e8a3c5be\" (UID: \"257ae2f6-4912-4362-b500-b886e8a3c5be\") " Jul 29 04:01:51.539394 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.539368 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-bundle" (OuterVolumeSpecName: "bundle") pod "257ae2f6-4912-4362-b500-b886e8a3c5be" (UID: "257ae2f6-4912-4362-b500-b886e8a3c5be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:51.541119 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.541097 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257ae2f6-4912-4362-b500-b886e8a3c5be-kube-api-access-xd8tv" (OuterVolumeSpecName: "kube-api-access-xd8tv") pod "257ae2f6-4912-4362-b500-b886e8a3c5be" (UID: "257ae2f6-4912-4362-b500-b886e8a3c5be"). InnerVolumeSpecName "kube-api-access-xd8tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:01:51.544324 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.544294 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-util" (OuterVolumeSpecName: "util") pod "257ae2f6-4912-4362-b500-b886e8a3c5be" (UID: "257ae2f6-4912-4362-b500-b886e8a3c5be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:01:51.639912 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.639869 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-util\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:51.640020 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.639916 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/257ae2f6-4912-4362-b500-b886e8a3c5be-bundle\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:51.640020 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:51.639930 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8tv\" (UniqueName: \"kubernetes.io/projected/257ae2f6-4912-4362-b500-b886e8a3c5be-kube-api-access-xd8tv\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:01:52.266934 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:52.266907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" event={"ID":"257ae2f6-4912-4362-b500-b886e8a3c5be","Type":"ContainerDied","Data":"07779d7606887f4940b00156204cd472875ff1c46772d7fd51c4ad3e89f0be61"} Jul 29 04:01:52.266934 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:52.266928 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z" Jul 29 04:01:52.267335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:52.266947 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07779d7606887f4940b00156204cd472875ff1c46772d7fd51c4ad3e89f0be61" Jul 29 04:01:58.326453 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:58.326424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:01:59.141533 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:01:59.141504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:03.739163 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:03.739125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:05.600400 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600367 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t"] Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600684 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerName="util" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600695 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerName="util" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600705 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerName="pull" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600710 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerName="pull" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600721 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerName="pull" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600726 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerName="pull" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600731 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerName="util" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600736 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerName="util" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600743 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerName="pull" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600747 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerName="pull" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600754 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerName="pull" Jul 29 04:02:05.600756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600759 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerName="pull" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600765 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerName="util" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600769 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerName="util" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600774 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600778 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600787 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600792 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600797 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600802 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600811 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600816 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600822 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerName="util" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600827 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerName="util" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600907 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="399cdbd9-8aa0-4f0d-8b29-191573cc1699" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600918 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="05790ad2-5d86-4ebd-b90c-83c79f96ff07" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600924 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a214469-3308-42d4-b967-9d9b36e96b3b" containerName="extract" Jul 29 04:02:05.601138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.600929 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="257ae2f6-4912-4362-b500-b886e8a3c5be" containerName="extract" Jul 29 04:02:05.604115 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.604093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" Jul 29 04:02:05.606655 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.606629 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 04:02:05.606979 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.606964 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 04:02:05.607176 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.607162 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 04:02:05.608080 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.608063 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7nxl2\"" Jul 29 04:02:05.618771 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.618750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t"] Jul 29 04:02:05.652297 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.652270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbndv\" (UniqueName: \"kubernetes.io/projected/5c99abd3-fb23-4110-b208-7b78f0fe89ff-kube-api-access-wbndv\") pod \"dns-operator-controller-manager-864dd78c4c-2ts5t\" (UID: \"5c99abd3-fb23-4110-b208-7b78f0fe89ff\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" Jul 29 04:02:05.753199 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.753170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbndv\" (UniqueName: \"kubernetes.io/projected/5c99abd3-fb23-4110-b208-7b78f0fe89ff-kube-api-access-wbndv\") pod \"dns-operator-controller-manager-864dd78c4c-2ts5t\" (UID: \"5c99abd3-fb23-4110-b208-7b78f0fe89ff\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" Jul 29 04:02:05.768716 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.768686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbndv\" (UniqueName: \"kubernetes.io/projected/5c99abd3-fb23-4110-b208-7b78f0fe89ff-kube-api-access-wbndv\") pod \"dns-operator-controller-manager-864dd78c4c-2ts5t\" (UID: \"5c99abd3-fb23-4110-b208-7b78f0fe89ff\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" Jul 29 04:02:05.784055 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.784035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:05.881078 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.881050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:05.914566 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:05.914543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" Jul 29 04:02:06.050560 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:06.050540 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t"] Jul 29 04:02:06.052277 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:02:06.052253 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c99abd3_fb23_4110_b208_7b78f0fe89ff.slice/crio-bb512b461ed3a4464fbacf8430d2bdb45b7b3839e151a98261b09faa47e0cced WatchSource:0}: Error finding container bb512b461ed3a4464fbacf8430d2bdb45b7b3839e151a98261b09faa47e0cced: Status 404 returned error can't find the container with id bb512b461ed3a4464fbacf8430d2bdb45b7b3839e151a98261b09faa47e0cced Jul 29 04:02:06.340999 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:06.340917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" event={"ID":"5c99abd3-fb23-4110-b208-7b78f0fe89ff","Type":"ContainerStarted","Data":"bb512b461ed3a4464fbacf8430d2bdb45b7b3839e151a98261b09faa47e0cced"} Jul 29 04:02:06.779839 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:06.779791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:07.583065 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:07.583028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:07.830758 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:07.830710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:09.128816 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.128794 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm"] Jul 29 04:02:09.132030 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.132013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:09.134522 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.134505 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6dj47\"" Jul 29 04:02:09.145931 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.145909 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm"] Jul 29 04:02:09.181368 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.181335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24znh\" (UniqueName: \"kubernetes.io/projected/e44f1361-404b-4b01-95e4-4613361ba326-kube-api-access-24znh\") pod \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" (UID: \"e44f1361-404b-4b01-95e4-4613361ba326\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:09.282724 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.282699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24znh\" (UniqueName: \"kubernetes.io/projected/e44f1361-404b-4b01-95e4-4613361ba326-kube-api-access-24znh\") pod \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" (UID: \"e44f1361-404b-4b01-95e4-4613361ba326\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:09.290919 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.290872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24znh\" (UniqueName: \"kubernetes.io/projected/e44f1361-404b-4b01-95e4-4613361ba326-kube-api-access-24znh\") pod \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" (UID: \"e44f1361-404b-4b01-95e4-4613361ba326\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:09.363177 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.363147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" event={"ID":"5c99abd3-fb23-4110-b208-7b78f0fe89ff","Type":"ContainerStarted","Data":"248b95f317e0389dd610a07243a15aa1253c34ec984201c1a1bd38d1f244b54b"} Jul 29 04:02:09.363280 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.363203 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" Jul 29 04:02:09.383851 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.383773 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" podStartSLOduration=1.770042085 podStartE2EDuration="4.383759299s" totalImagesPullingTime="2.613717214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:02:09.381403989 +0000 UTC m=+411.673500350" watchObservedRunningTime="2026-07-29 04:02:09.383759299 +0000 UTC m=+411.675855658" Jul 29 04:02:09.442843 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.442821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:09.581130 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.581106 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm"] Jul 29 04:02:09.582791 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:02:09.582765 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44f1361_404b_4b01_95e4_4613361ba326.slice/crio-77d9386327311328d1c71227cfec59f7954bd3c0b43607fbffa532b3bb81154e WatchSource:0}: Error finding container 77d9386327311328d1c71227cfec59f7954bd3c0b43607fbffa532b3bb81154e: Status 404 returned error can't find the container with id 77d9386327311328d1c71227cfec59f7954bd3c0b43607fbffa532b3bb81154e Jul 29 04:02:09.806758 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:09.806663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:10.188162 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:10.188124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:10.371194 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:10.371164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" event={"ID":"e44f1361-404b-4b01-95e4-4613361ba326","Type":"ContainerStarted","Data":"77d9386327311328d1c71227cfec59f7954bd3c0b43607fbffa532b3bb81154e"} Jul 29 04:02:12.382402 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:12.382368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" event={"ID":"e44f1361-404b-4b01-95e4-4613361ba326","Type":"ContainerStarted","Data":"fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814"} Jul 29 04:02:12.382805 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:12.382430 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:12.402262 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:12.402214 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" podStartSLOduration=1.5992356399999998 podStartE2EDuration="3.402199463s" totalImagesPullingTime="1.802963823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:02:12.400502588 +0000 UTC m=+414.692598951" watchObservedRunningTime="2026-07-29 04:02:12.402199463 +0000 UTC m=+414.694295875" Jul 29 04:02:13.413764 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:13.413728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:15.348841 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:15.348807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:15.447723 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:15.447685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:15.751283 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:15.751249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:15.850686 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:15.850657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:18.379958 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:18.379926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:18.479910 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:18.479859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:19.147855 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:19.147821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:19.247672 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:19.247634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:20.373516 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:20.373491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2ts5t" Jul 29 04:02:22.249381 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:22.249345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:22.709781 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:22.709739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:23.349699 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:23.349653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:23.389860 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:23.389836 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:23.450400 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:23.450368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:26.469467 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:26.469434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:26.566500 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:26.566466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:32.565262 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.565226 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc"] Jul 29 04:02:32.569080 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.569059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.571540 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.571518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pc9k8\"" Jul 29 04:02:32.578910 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.578807 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm"] Jul 29 04:02:32.579073 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.579026 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" podUID="e44f1361-404b-4b01-95e4-4613361ba326" containerName="manager" containerID="cri-o://fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814" gracePeriod=2 Jul 29 04:02:32.584109 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.584077 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc"] Jul 29 04:02:32.591494 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.591471 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm"] Jul 29 04:02:32.599295 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:02:32.599268 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" is forbidden: User \"system:node:ip-10-0-140-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-47.ec2.internal' and this object" podUID="e44f1361-404b-4b01-95e4-4613361ba326" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:32.604417 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.604396 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz"] Jul 29 04:02:32.604736 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.604725 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44f1361-404b-4b01-95e4-4613361ba326" containerName="manager" Jul 29 04:02:32.604777 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.604739 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44f1361-404b-4b01-95e4-4613361ba326" containerName="manager" Jul 29 04:02:32.604842 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.604792 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44f1361-404b-4b01-95e4-4613361ba326" containerName="manager" Jul 29 04:02:32.607954 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.607938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" Jul 29 04:02:32.610355 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:02:32.610274 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" is forbidden: User \"system:node:ip-10-0-140-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-47.ec2.internal' and this object" podUID="e44f1361-404b-4b01-95e4-4613361ba326" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:32.617080 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.617056 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz"] Jul 29 04:02:32.673073 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.673046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mw28\" (UniqueName: \"kubernetes.io/projected/55b21408-7f04-4278-96d3-90937e9b61a1-kube-api-access-7mw28\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vrklc\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.673188 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.673085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55b21408-7f04-4278-96d3-90937e9b61a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vrklc\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.774663 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.774625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mw28\" (UniqueName: \"kubernetes.io/projected/55b21408-7f04-4278-96d3-90937e9b61a1-kube-api-access-7mw28\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vrklc\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.774807 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.774710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55b21408-7f04-4278-96d3-90937e9b61a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vrklc\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.774807 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.774752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chfdw\" (UniqueName: \"kubernetes.io/projected/a966ac0e-eeb3-41a8-bf97-fc98c424c8c8-kube-api-access-chfdw\") pod \"limitador-operator-controller-manager-859d7b78bd-ztwbz\" (UID: \"a966ac0e-eeb3-41a8-bf97-fc98c424c8c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" Jul 29 04:02:32.775125 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.775107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55b21408-7f04-4278-96d3-90937e9b61a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vrklc\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.788499 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.788478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mw28\" (UniqueName: \"kubernetes.io/projected/55b21408-7f04-4278-96d3-90937e9b61a1-kube-api-access-7mw28\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vrklc\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.839473 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.839448 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:32.841742 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:02:32.841720 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" is forbidden: User \"system:node:ip-10-0-140-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-47.ec2.internal' and this object" podUID="e44f1361-404b-4b01-95e4-4613361ba326" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:32.875600 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.875575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chfdw\" (UniqueName: \"kubernetes.io/projected/a966ac0e-eeb3-41a8-bf97-fc98c424c8c8-kube-api-access-chfdw\") pod \"limitador-operator-controller-manager-859d7b78bd-ztwbz\" (UID: \"a966ac0e-eeb3-41a8-bf97-fc98c424c8c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" Jul 29 04:02:32.881950 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.881932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:32.888163 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.888136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chfdw\" (UniqueName: \"kubernetes.io/projected/a966ac0e-eeb3-41a8-bf97-fc98c424c8c8-kube-api-access-chfdw\") pod \"limitador-operator-controller-manager-859d7b78bd-ztwbz\" (UID: \"a966ac0e-eeb3-41a8-bf97-fc98c424c8c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" Jul 29 04:02:32.963178 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.963148 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" Jul 29 04:02:32.976798 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.976772 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24znh\" (UniqueName: \"kubernetes.io/projected/e44f1361-404b-4b01-95e4-4613361ba326-kube-api-access-24znh\") pod \"e44f1361-404b-4b01-95e4-4613361ba326\" (UID: \"e44f1361-404b-4b01-95e4-4613361ba326\") " Jul 29 04:02:32.980107 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:32.980077 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44f1361-404b-4b01-95e4-4613361ba326-kube-api-access-24znh" (OuterVolumeSpecName: "kube-api-access-24znh") pod "e44f1361-404b-4b01-95e4-4613361ba326" (UID: "e44f1361-404b-4b01-95e4-4613361ba326"). InnerVolumeSpecName "kube-api-access-24znh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:02:33.029495 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.029428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc"] Jul 29 04:02:33.031930 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:02:33.031872 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b21408_7f04_4278_96d3_90937e9b61a1.slice/crio-9f5d1bd1c62e339c53ee49bc9949c73fb4c827cccef6a8142f06a05b8ff1c892 WatchSource:0}: Error finding container 9f5d1bd1c62e339c53ee49bc9949c73fb4c827cccef6a8142f06a05b8ff1c892: Status 404 returned error can't find the container with id 9f5d1bd1c62e339c53ee49bc9949c73fb4c827cccef6a8142f06a05b8ff1c892 Jul 29 04:02:33.077658 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.077592 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24znh\" (UniqueName: \"kubernetes.io/projected/e44f1361-404b-4b01-95e4-4613361ba326-kube-api-access-24znh\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:02:33.101825 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.101803 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz"] Jul 29 04:02:33.104249 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:02:33.104224 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda966ac0e_eeb3_41a8_bf97_fc98c424c8c8.slice/crio-3b193854252d3b484a065e2fbe579b425a97927ec48dd724c7192da9df1c60fe WatchSource:0}: Error finding container 3b193854252d3b484a065e2fbe579b425a97927ec48dd724c7192da9df1c60fe: Status 404 returned error can't find the container with id 3b193854252d3b484a065e2fbe579b425a97927ec48dd724c7192da9df1c60fe Jul 29 04:02:33.505760 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.505722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" event={"ID":"a966ac0e-eeb3-41a8-bf97-fc98c424c8c8","Type":"ContainerStarted","Data":"bdadcf3a0cb2954870bd50a84db122ba907f66d17d85141afac2d15fe363b6cb"} Jul 29 04:02:33.505958 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.505797 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" Jul 29 04:02:33.505958 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.505821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" event={"ID":"a966ac0e-eeb3-41a8-bf97-fc98c424c8c8","Type":"ContainerStarted","Data":"3b193854252d3b484a065e2fbe579b425a97927ec48dd724c7192da9df1c60fe"} Jul 29 04:02:33.507124 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.507098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" event={"ID":"55b21408-7f04-4278-96d3-90937e9b61a1","Type":"ContainerStarted","Data":"9f5d1bd1c62e339c53ee49bc9949c73fb4c827cccef6a8142f06a05b8ff1c892"} Jul 29 04:02:33.508383 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.508357 2576 generic.go:356] "Generic (PLEG): container finished" podID="e44f1361-404b-4b01-95e4-4613361ba326" containerID="fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814" exitCode=0 Jul 29 04:02:33.508470 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.508396 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:33.508470 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.508404 2576 scope.go:117] "RemoveContainer" containerID="fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814" Jul 29 04:02:33.522895 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.522854 2576 scope.go:117] "RemoveContainer" containerID="fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814" Jul 29 04:02:33.523182 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:02:33.523151 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814\": container with ID starting with fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814 not found: ID does not exist" containerID="fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814" Jul 29 04:02:33.523290 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.523197 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814"} err="failed to get container status \"fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814\": rpc error: code = NotFound desc = could not find container \"fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814\": container with ID starting with fc67f16d68155c93775245970ed7148c466c8183471bc25006bcbbb89e011814 not found: ID does not exist" Jul 29 04:02:33.526555 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:02:33.526523 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" is forbidden: User \"system:node:ip-10-0-140-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-47.ec2.internal' and this object" podUID="e44f1361-404b-4b01-95e4-4613361ba326" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:33.526971 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:33.526932 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" podStartSLOduration=1.526917783 podStartE2EDuration="1.526917783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:02:33.524406828 +0000 UTC m=+435.816503183" watchObservedRunningTime="2026-07-29 04:02:33.526917783 +0000 UTC m=+435.819014204" Jul 29 04:02:33.528529 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:02:33.528503 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rqgnm\" is forbidden: User \"system:node:ip-10-0-140-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-47.ec2.internal' and this object" podUID="e44f1361-404b-4b01-95e4-4613361ba326" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rqgnm" Jul 29 04:02:34.201545 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:34.201507 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44f1361-404b-4b01-95e4-4613361ba326" path="/var/lib/kubelet/pods/e44f1361-404b-4b01-95e4-4613361ba326/volumes" Jul 29 04:02:34.505386 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:34.505274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:34.605773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:34.605742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:35.108014 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:35.107985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:35.204574 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:35.204538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:37.534396 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:37.534363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" event={"ID":"55b21408-7f04-4278-96d3-90937e9b61a1","Type":"ContainerStarted","Data":"0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81"} Jul 29 04:02:37.534939 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:37.534472 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:02:37.554770 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:37.554726 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" podStartSLOduration=2.05000026 podStartE2EDuration="5.554713731s" totalImagesPullingTime="3.504713471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:02:37.55320173 +0000 UTC m=+439.845298166" watchObservedRunningTime="2026-07-29 04:02:37.554713731 +0000 UTC m=+439.846810092" Jul 29 04:02:38.090508 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:38.090468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:38.142871 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:38.142836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:42.695607 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:42.695520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:42.807609 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:42.807580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:44.518230 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:44.518200 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ztwbz" Jul 29 04:02:45.292155 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:45.292113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:45.391182 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:45.391149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:02:48.541803 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:02:48.541768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:03:01.044797 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.044753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc"] Jul 29 04:03:01.045328 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.045016 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" podUID="55b21408-7f04-4278-96d3-90937e9b61a1" containerName="manager" containerID="cri-o://0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81" gracePeriod=10 Jul 29 04:03:01.308118 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.308061 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:03:01.395246 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.395212 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55b21408-7f04-4278-96d3-90937e9b61a1-extensions-socket-volume\") pod \"55b21408-7f04-4278-96d3-90937e9b61a1\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " Jul 29 04:03:01.395410 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.395281 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mw28\" (UniqueName: \"kubernetes.io/projected/55b21408-7f04-4278-96d3-90937e9b61a1-kube-api-access-7mw28\") pod \"55b21408-7f04-4278-96d3-90937e9b61a1\" (UID: \"55b21408-7f04-4278-96d3-90937e9b61a1\") " Jul 29 04:03:01.395735 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.395712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b21408-7f04-4278-96d3-90937e9b61a1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "55b21408-7f04-4278-96d3-90937e9b61a1" (UID: "55b21408-7f04-4278-96d3-90937e9b61a1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:03:01.397515 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.397494 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b21408-7f04-4278-96d3-90937e9b61a1-kube-api-access-7mw28" (OuterVolumeSpecName: "kube-api-access-7mw28") pod "55b21408-7f04-4278-96d3-90937e9b61a1" (UID: "55b21408-7f04-4278-96d3-90937e9b61a1"). InnerVolumeSpecName "kube-api-access-7mw28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:03:01.496020 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.495984 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mw28\" (UniqueName: \"kubernetes.io/projected/55b21408-7f04-4278-96d3-90937e9b61a1-kube-api-access-7mw28\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:03:01.496020 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.496011 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55b21408-7f04-4278-96d3-90937e9b61a1-extensions-socket-volume\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:03:01.663429 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.663391 2576 generic.go:356] "Generic (PLEG): container finished" podID="55b21408-7f04-4278-96d3-90937e9b61a1" containerID="0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81" exitCode=0 Jul 29 04:03:01.663611 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.663433 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" Jul 29 04:03:01.663611 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.663469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" event={"ID":"55b21408-7f04-4278-96d3-90937e9b61a1","Type":"ContainerDied","Data":"0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81"} Jul 29 04:03:01.663611 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.663531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc" event={"ID":"55b21408-7f04-4278-96d3-90937e9b61a1","Type":"ContainerDied","Data":"9f5d1bd1c62e339c53ee49bc9949c73fb4c827cccef6a8142f06a05b8ff1c892"} Jul 29 04:03:01.663611 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.663557 2576 scope.go:117] "RemoveContainer" containerID="0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81" Jul 29 04:03:01.673566 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.673549 2576 scope.go:117] "RemoveContainer" containerID="0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81" Jul 29 04:03:01.673828 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:03:01.673809 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81\": container with ID starting with 0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81 not found: ID does not exist" containerID="0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81" Jul 29 04:03:01.673907 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.673839 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81"} err="failed to get container status \"0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81\": rpc error: code = NotFound desc = could not find container \"0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81\": container with ID starting with 0dbb0752936f3e0f4fca93784c8d7ff521fedc6abac9611365163fe0b9878c81 not found: ID does not exist" Jul 29 04:03:01.685742 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.685720 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc"] Jul 29 04:03:01.689507 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:01.689485 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vrklc"] Jul 29 04:03:02.196700 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:02.196661 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b21408-7f04-4278-96d3-90937e9b61a1" path="/var/lib/kubelet/pods/55b21408-7f04-4278-96d3-90937e9b61a1/volumes" Jul 29 04:03:04.867771 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:04.867731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:05.466306 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:05.466273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:08.355798 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.355759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:08.362516 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.362487 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dxwv9"] Jul 29 04:03:08.362837 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.362825 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b21408-7f04-4278-96d3-90937e9b61a1" containerName="manager" Jul 29 04:03:08.362897 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.362839 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b21408-7f04-4278-96d3-90937e9b61a1" containerName="manager" Jul 29 04:03:08.362936 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.362926 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b21408-7f04-4278-96d3-90937e9b61a1" containerName="manager" Jul 29 04:03:08.365985 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.365967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.368448 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.368430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jbt2t\"" Jul 29 04:03:08.368567 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.368433 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 04:03:08.375100 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.375081 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dxwv9"] Jul 29 04:03:08.444048 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.444018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6m9\" (UniqueName: \"kubernetes.io/projected/36a0da79-0145-4d90-8ac1-2e6083e9a81c-kube-api-access-mx6m9\") pod \"limitador-limitador-64c94fd64f-dxwv9\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.444213 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.444069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/36a0da79-0145-4d90-8ac1-2e6083e9a81c-config-file\") pod \"limitador-limitador-64c94fd64f-dxwv9\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.463925 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.463874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dxwv9"] Jul 29 04:03:08.490058 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.490027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:08.545022 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.544994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6m9\" (UniqueName: \"kubernetes.io/projected/36a0da79-0145-4d90-8ac1-2e6083e9a81c-kube-api-access-mx6m9\") pod \"limitador-limitador-64c94fd64f-dxwv9\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.545163 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.545035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/36a0da79-0145-4d90-8ac1-2e6083e9a81c-config-file\") pod \"limitador-limitador-64c94fd64f-dxwv9\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.545623 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.545606 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/36a0da79-0145-4d90-8ac1-2e6083e9a81c-config-file\") pod \"limitador-limitador-64c94fd64f-dxwv9\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.548090 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.548063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:08.553007 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.552988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6m9\" (UniqueName: \"kubernetes.io/projected/36a0da79-0145-4d90-8ac1-2e6083e9a81c-kube-api-access-mx6m9\") pod \"limitador-limitador-64c94fd64f-dxwv9\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.677383 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.677355 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:08.814003 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:08.813977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dxwv9"] Jul 29 04:03:08.815409 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:03:08.815381 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a0da79_0145_4d90_8ac1_2e6083e9a81c.slice/crio-2a06506717bfb600c65e33c9c17c1e30f147af00b78d29433b5200a89b974201 WatchSource:0}: Error finding container 2a06506717bfb600c65e33c9c17c1e30f147af00b78d29433b5200a89b974201: Status 404 returned error can't find the container with id 2a06506717bfb600c65e33c9c17c1e30f147af00b78d29433b5200a89b974201 Jul 29 04:03:09.272363 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:09.272334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:09.337326 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:09.337293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:09.711715 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:09.711678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" event={"ID":"36a0da79-0145-4d90-8ac1-2e6083e9a81c","Type":"ContainerStarted","Data":"2a06506717bfb600c65e33c9c17c1e30f147af00b78d29433b5200a89b974201"} Jul 29 04:03:12.728480 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:12.728449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" event={"ID":"36a0da79-0145-4d90-8ac1-2e6083e9a81c","Type":"ContainerStarted","Data":"b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619"} Jul 29 04:03:12.728851 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:12.728560 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:12.746683 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:12.746630 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" podStartSLOduration=1.893903086 podStartE2EDuration="4.746613025s" totalImagesPullingTime="2.852709939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:12.745440188 +0000 UTC m=+475.037536561" watchObservedRunningTime="2026-07-29 04:03:12.746613025 +0000 UTC m=+475.038709388" Jul 29 04:03:15.128463 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:15.128418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:15.190148 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:15.190122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:23.734618 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:23.734587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:23.914996 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:23.914946 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dxwv9"] Jul 29 04:03:23.915236 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:23.915188 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" podUID="36a0da79-0145-4d90-8ac1-2e6083e9a81c" containerName="limitador" containerID="cri-o://b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619" gracePeriod=30 Jul 29 04:03:24.475099 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.475033 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:24.577907 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.577854 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx6m9\" (UniqueName: \"kubernetes.io/projected/36a0da79-0145-4d90-8ac1-2e6083e9a81c-kube-api-access-mx6m9\") pod \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " Jul 29 04:03:24.578092 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.577969 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/36a0da79-0145-4d90-8ac1-2e6083e9a81c-config-file\") pod \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\" (UID: \"36a0da79-0145-4d90-8ac1-2e6083e9a81c\") " Jul 29 04:03:24.578336 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.578306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a0da79-0145-4d90-8ac1-2e6083e9a81c-config-file" (OuterVolumeSpecName: "config-file") pod "36a0da79-0145-4d90-8ac1-2e6083e9a81c" (UID: "36a0da79-0145-4d90-8ac1-2e6083e9a81c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:03:24.580060 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.580032 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a0da79-0145-4d90-8ac1-2e6083e9a81c-kube-api-access-mx6m9" (OuterVolumeSpecName: "kube-api-access-mx6m9") pod "36a0da79-0145-4d90-8ac1-2e6083e9a81c" (UID: "36a0da79-0145-4d90-8ac1-2e6083e9a81c"). InnerVolumeSpecName "kube-api-access-mx6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:03:24.679085 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.679054 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/36a0da79-0145-4d90-8ac1-2e6083e9a81c-config-file\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:03:24.679085 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.679083 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6m9\" (UniqueName: \"kubernetes.io/projected/36a0da79-0145-4d90-8ac1-2e6083e9a81c-kube-api-access-mx6m9\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:03:24.792947 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.792829 2576 generic.go:356] "Generic (PLEG): container finished" podID="36a0da79-0145-4d90-8ac1-2e6083e9a81c" containerID="b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619" exitCode=0 Jul 29 04:03:24.792947 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.792898 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" Jul 29 04:03:24.792947 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.792919 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" event={"ID":"36a0da79-0145-4d90-8ac1-2e6083e9a81c","Type":"ContainerDied","Data":"b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619"} Jul 29 04:03:24.793412 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.792970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dxwv9" event={"ID":"36a0da79-0145-4d90-8ac1-2e6083e9a81c","Type":"ContainerDied","Data":"2a06506717bfb600c65e33c9c17c1e30f147af00b78d29433b5200a89b974201"} Jul 29 04:03:24.793412 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.792992 2576 scope.go:117] "RemoveContainer" containerID="b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619" Jul 29 04:03:24.802596 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.802580 2576 scope.go:117] "RemoveContainer" containerID="b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619" Jul 29 04:03:24.802839 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:03:24.802820 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619\": container with ID starting with b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619 not found: ID does not exist" containerID="b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619" Jul 29 04:03:24.802945 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.802852 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619"} err="failed to get container status \"b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619\": rpc error: code = NotFound desc = could not find container \"b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619\": container with ID starting with b19c3d8c6e3eff38fe35d793cb021f2babd3069c5c1599bc88a4f2387fee4619 not found: ID does not exist" Jul 29 04:03:24.814905 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.814872 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dxwv9"] Jul 29 04:03:24.820169 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:24.820138 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dxwv9"] Jul 29 04:03:26.196766 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:26.196731 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a0da79-0145-4d90-8ac1-2e6083e9a81c" path="/var/lib/kubelet/pods/36a0da79-0145-4d90-8ac1-2e6083e9a81c/volumes" Jul 29 04:03:27.700295 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:27.700262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:27.760338 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:27.760305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:28.012676 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.012584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:28.110500 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.110468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:28.481930 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.481867 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq"] Jul 29 04:03:28.482440 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.482426 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a0da79-0145-4d90-8ac1-2e6083e9a81c" containerName="limitador" Jul 29 04:03:28.482485 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.482444 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a0da79-0145-4d90-8ac1-2e6083e9a81c" containerName="limitador" Jul 29 04:03:28.482545 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.482534 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a0da79-0145-4d90-8ac1-2e6083e9a81c" containerName="limitador" Jul 29 04:03:28.485767 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.485748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.489411 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.489385 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 04:03:28.489411 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.489397 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2fqjj\"" Jul 29 04:03:28.489569 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.489452 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 04:03:28.490285 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.490268 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 04:03:28.503365 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.503348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq"] Jul 29 04:03:28.537255 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.537234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:28.612115 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.612092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25caae94-4fd5-4a86-9f79-4730a99c1cea-data\") pod \"postgres-69fb4594cf-vwhnq\" (UID: \"25caae94-4fd5-4a86-9f79-4730a99c1cea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.612230 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.612123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqhh\" (UniqueName: \"kubernetes.io/projected/25caae94-4fd5-4a86-9f79-4730a99c1cea-kube-api-access-tdqhh\") pod \"postgres-69fb4594cf-vwhnq\" (UID: \"25caae94-4fd5-4a86-9f79-4730a99c1cea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.712960 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.712930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25caae94-4fd5-4a86-9f79-4730a99c1cea-data\") pod \"postgres-69fb4594cf-vwhnq\" (UID: \"25caae94-4fd5-4a86-9f79-4730a99c1cea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.712960 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.712964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqhh\" (UniqueName: \"kubernetes.io/projected/25caae94-4fd5-4a86-9f79-4730a99c1cea-kube-api-access-tdqhh\") pod \"postgres-69fb4594cf-vwhnq\" (UID: \"25caae94-4fd5-4a86-9f79-4730a99c1cea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.713375 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.713355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25caae94-4fd5-4a86-9f79-4730a99c1cea-data\") pod \"postgres-69fb4594cf-vwhnq\" (UID: \"25caae94-4fd5-4a86-9f79-4730a99c1cea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.721931 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.721871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqhh\" (UniqueName: \"kubernetes.io/projected/25caae94-4fd5-4a86-9f79-4730a99c1cea-kube-api-access-tdqhh\") pod \"postgres-69fb4594cf-vwhnq\" (UID: \"25caae94-4fd5-4a86-9f79-4730a99c1cea\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.796287 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.796185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:28.926371 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:28.926339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq"] Jul 29 04:03:28.927640 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:03:28.927614 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25caae94_4fd5_4a86_9f79_4730a99c1cea.slice/crio-209cd08774ff018bf99d786e7232e1788d657556d3e86f3ea6b0c458475c01d3 WatchSource:0}: Error finding container 209cd08774ff018bf99d786e7232e1788d657556d3e86f3ea6b0c458475c01d3: Status 404 returned error can't find the container with id 209cd08774ff018bf99d786e7232e1788d657556d3e86f3ea6b0c458475c01d3 Jul 29 04:03:29.823926 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:29.823865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" event={"ID":"25caae94-4fd5-4a86-9f79-4730a99c1cea","Type":"ContainerStarted","Data":"209cd08774ff018bf99d786e7232e1788d657556d3e86f3ea6b0c458475c01d3"} Jul 29 04:03:34.857004 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:34.856972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" event={"ID":"25caae94-4fd5-4a86-9f79-4730a99c1cea","Type":"ContainerStarted","Data":"23d12673707a3a896ce2b89ced9340a2bee04464456cb50cec484d51f5151a14"} Jul 29 04:03:34.857391 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:34.857084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:34.873685 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:34.873637 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" podStartSLOduration=1.93321881 podStartE2EDuration="6.873622585s" totalImagesPullingTime="4.940403775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:34.872109732 +0000 UTC m=+497.164206097" watchObservedRunningTime="2026-07-29 04:03:34.873622585 +0000 UTC m=+497.165718946" Jul 29 04:03:38.385789 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:38.385754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:38.483571 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:38.483538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:38.584407 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:38.584366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:40.891564 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:40.891538 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vwhnq" Jul 29 04:03:41.206677 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.206578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:41.373919 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.373873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:41.656383 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.656355 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-knqqx"] Jul 29 04:03:41.659994 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.659976 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.663670 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.663644 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 04:03:41.663779 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.663711 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dht7f\"" Jul 29 04:03:41.670240 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.670222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-knqqx"] Jul 29 04:03:41.698323 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.698299 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-knqqx"] Jul 29 04:03:41.698591 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:03:41.698566 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ptl4r tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-knqqx" podUID="80d475fe-c0c5-41ca-98af-71a45a740a97" Jul 29 04:03:41.721401 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.721374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80d475fe-c0c5-41ca-98af-71a45a740a97-tls-cert\") pod \"authorino-76877994-knqqx\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.721401 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.721403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptl4r\" (UniqueName: \"kubernetes.io/projected/80d475fe-c0c5-41ca-98af-71a45a740a97-kube-api-access-ptl4r\") pod \"authorino-76877994-knqqx\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.822133 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.822104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80d475fe-c0c5-41ca-98af-71a45a740a97-tls-cert\") pod \"authorino-76877994-knqqx\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.822273 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.822140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptl4r\" (UniqueName: \"kubernetes.io/projected/80d475fe-c0c5-41ca-98af-71a45a740a97-kube-api-access-ptl4r\") pod \"authorino-76877994-knqqx\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.824797 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.824776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80d475fe-c0c5-41ca-98af-71a45a740a97-tls-cert\") pod \"authorino-76877994-knqqx\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.833940 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.833913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptl4r\" (UniqueName: \"kubernetes.io/projected/80d475fe-c0c5-41ca-98af-71a45a740a97-kube-api-access-ptl4r\") pod \"authorino-76877994-knqqx\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.896904 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.895524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:41.906606 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:41.906549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:42.023420 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.023390 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80d475fe-c0c5-41ca-98af-71a45a740a97-tls-cert\") pod \"80d475fe-c0c5-41ca-98af-71a45a740a97\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " Jul 29 04:03:42.023612 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.023486 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptl4r\" (UniqueName: \"kubernetes.io/projected/80d475fe-c0c5-41ca-98af-71a45a740a97-kube-api-access-ptl4r\") pod \"80d475fe-c0c5-41ca-98af-71a45a740a97\" (UID: \"80d475fe-c0c5-41ca-98af-71a45a740a97\") " Jul 29 04:03:42.025789 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.025754 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d475fe-c0c5-41ca-98af-71a45a740a97-kube-api-access-ptl4r" (OuterVolumeSpecName: "kube-api-access-ptl4r") pod "80d475fe-c0c5-41ca-98af-71a45a740a97" (UID: "80d475fe-c0c5-41ca-98af-71a45a740a97"). InnerVolumeSpecName "kube-api-access-ptl4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:03:42.025789 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.025776 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d475fe-c0c5-41ca-98af-71a45a740a97-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "80d475fe-c0c5-41ca-98af-71a45a740a97" (UID: "80d475fe-c0c5-41ca-98af-71a45a740a97"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:03:42.124437 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.124403 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptl4r\" (UniqueName: \"kubernetes.io/projected/80d475fe-c0c5-41ca-98af-71a45a740a97-kube-api-access-ptl4r\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:03:42.124437 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.124429 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80d475fe-c0c5-41ca-98af-71a45a740a97-tls-cert\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:03:42.901776 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.901746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-knqqx" Jul 29 04:03:42.933690 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.933664 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-knqqx"] Jul 29 04:03:42.935613 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:42.935594 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-knqqx"] Jul 29 04:03:44.198440 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:44.198403 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80d475fe-c0c5-41ca-98af-71a45a740a97" path="/var/lib/kubelet/pods/80d475fe-c0c5-41ca-98af-71a45a740a97/volumes" Jul 29 04:03:45.023040 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:45.023013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:50.610991 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.610938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:50.611494 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.611475 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-569bbdd56d-n9ts8"] Jul 29 04:03:50.631626 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.631600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569bbdd56d-n9ts8"] Jul 29 04:03:50.631791 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.631693 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:50.636281 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.636257 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 04:03:50.636416 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.636306 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 04:03:50.636416 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.636353 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-t2j64\"" Jul 29 04:03:50.695673 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.695647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert\") pod \"maas-controller-569bbdd56d-n9ts8\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:50.695828 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.695727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zz2w\" (UniqueName: \"kubernetes.io/projected/ee09a0fe-af38-40ec-a928-473663d09d0f-kube-api-access-4zz2w\") pod \"maas-controller-569bbdd56d-n9ts8\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:50.700215 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.700172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:50.779131 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.779098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:50.788905 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.788859 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8cdf94955-j6v9p"] Jul 29 04:03:50.791339 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.791323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:50.796593 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.796572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert\") pod \"maas-controller-569bbdd56d-n9ts8\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:50.796710 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.796648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zz2w\" (UniqueName: \"kubernetes.io/projected/ee09a0fe-af38-40ec-a928-473663d09d0f-kube-api-access-4zz2w\") pod \"maas-controller-569bbdd56d-n9ts8\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:50.796777 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:03:50.796729 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 29 04:03:50.796831 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:03:50.796798 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert podName:ee09a0fe-af38-40ec-a928-473663d09d0f nodeName:}" failed. No retries permitted until 2026-07-29 04:03:51.296780652 +0000 UTC m=+513.588876998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert") pod "maas-controller-569bbdd56d-n9ts8" (UID: "ee09a0fe-af38-40ec-a928-473663d09d0f") : secret "maas-controller-webhook-cert" not found Jul 29 04:03:50.809546 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.809525 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cdf94955-j6v9p"] Jul 29 04:03:50.826525 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.826503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zz2w\" (UniqueName: \"kubernetes.io/projected/ee09a0fe-af38-40ec-a928-473663d09d0f-kube-api-access-4zz2w\") pod \"maas-controller-569bbdd56d-n9ts8\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:50.897410 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.897378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdhh\" (UniqueName: \"kubernetes.io/projected/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-kube-api-access-bhdhh\") pod \"maas-controller-8cdf94955-j6v9p\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:50.897576 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.897496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-webhook-cert\") pod \"maas-controller-8cdf94955-j6v9p\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:50.998389 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.998358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-webhook-cert\") pod \"maas-controller-8cdf94955-j6v9p\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:50.998573 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:50.998425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdhh\" (UniqueName: \"kubernetes.io/projected/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-kube-api-access-bhdhh\") pod \"maas-controller-8cdf94955-j6v9p\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:51.000933 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.000901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:51.001231 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.001213 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-webhook-cert\") pod \"maas-controller-8cdf94955-j6v9p\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:51.010038 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.010009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdhh\" (UniqueName: \"kubernetes.io/projected/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-kube-api-access-bhdhh\") pod \"maas-controller-8cdf94955-j6v9p\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:51.101957 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.101933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:51.245995 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.245974 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cdf94955-j6v9p"] Jul 29 04:03:51.247361 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:03:51.247337 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f9f794_6bca_4944_bdc2_a69e9d7d5c2e.slice/crio-d9c660136ca53dcd65715fb139c641b585b33228ddc53acb0e768df5b3b17e31 WatchSource:0}: Error finding container d9c660136ca53dcd65715fb139c641b585b33228ddc53acb0e768df5b3b17e31: Status 404 returned error can't find the container with id d9c660136ca53dcd65715fb139c641b585b33228ddc53acb0e768df5b3b17e31 Jul 29 04:03:51.300048 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.300024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert\") pod \"maas-controller-569bbdd56d-n9ts8\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:51.302318 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.302300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert\") pod \"maas-controller-569bbdd56d-n9ts8\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:51.542915 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.542838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:51.678239 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.678217 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569bbdd56d-n9ts8"] Jul 29 04:03:51.679445 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:03:51.679422 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee09a0fe_af38_40ec_a928_473663d09d0f.slice/crio-022662cdbfb26589b9936e726a7ee4f465ab5ad0c43a3ce340e619e68858be02 WatchSource:0}: Error finding container 022662cdbfb26589b9936e726a7ee4f465ab5ad0c43a3ce340e619e68858be02: Status 404 returned error can't find the container with id 022662cdbfb26589b9936e726a7ee4f465ab5ad0c43a3ce340e619e68858be02 Jul 29 04:03:51.953289 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.953252 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" event={"ID":"ee09a0fe-af38-40ec-a928-473663d09d0f","Type":"ContainerStarted","Data":"022662cdbfb26589b9936e726a7ee4f465ab5ad0c43a3ce340e619e68858be02"} Jul 29 04:03:51.954358 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:51.954336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cdf94955-j6v9p" event={"ID":"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e","Type":"ContainerStarted","Data":"d9c660136ca53dcd65715fb139c641b585b33228ddc53acb0e768df5b3b17e31"} Jul 29 04:03:54.374412 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:54.374382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:54.476196 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:54.476112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:03:54.974310 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:54.974268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" event={"ID":"ee09a0fe-af38-40ec-a928-473663d09d0f","Type":"ContainerStarted","Data":"43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a"} Jul 29 04:03:54.974506 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:54.974376 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:03:54.975992 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:54.975967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cdf94955-j6v9p" event={"ID":"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e","Type":"ContainerStarted","Data":"aabd305fe45a38aa071792a220333e90df3d48798c492a8d2011a6dfc5faf47f"} Jul 29 04:03:54.976135 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:54.976022 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:03:55.001112 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:55.001064 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" podStartSLOduration=2.482766691 podStartE2EDuration="5.001047742s" totalImagesPullingTime="2.518281051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:55.000329682 +0000 UTC m=+517.292426070" watchObservedRunningTime="2026-07-29 04:03:55.001047742 +0000 UTC m=+517.293144106" Jul 29 04:03:55.047311 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:03:55.047261 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8cdf94955-j6v9p" podStartSLOduration=2.090394974 podStartE2EDuration="5.047248871s" totalImagesPullingTime="2.956853897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:55.04514454 +0000 UTC m=+517.337240902" watchObservedRunningTime="2026-07-29 04:03:55.047248871 +0000 UTC m=+517.339345239" Jul 29 04:04:05.985100 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:05.985070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:04:05.985460 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:05.985141 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:04:06.047066 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.047027 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569bbdd56d-n9ts8"] Jul 29 04:04:06.047244 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.047206 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" podUID="ee09a0fe-af38-40ec-a928-473663d09d0f" containerName="manager" containerID="cri-o://43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a" gracePeriod=30 Jul 29 04:04:06.300843 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.300818 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:04:06.433664 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.433629 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zz2w\" (UniqueName: \"kubernetes.io/projected/ee09a0fe-af38-40ec-a928-473663d09d0f-kube-api-access-4zz2w\") pod \"ee09a0fe-af38-40ec-a928-473663d09d0f\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " Jul 29 04:04:06.433838 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.433684 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert\") pod \"ee09a0fe-af38-40ec-a928-473663d09d0f\" (UID: \"ee09a0fe-af38-40ec-a928-473663d09d0f\") " Jul 29 04:04:06.436054 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.436024 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ee09a0fe-af38-40ec-a928-473663d09d0f" (UID: "ee09a0fe-af38-40ec-a928-473663d09d0f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:04:06.436139 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.436049 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee09a0fe-af38-40ec-a928-473663d09d0f-kube-api-access-4zz2w" (OuterVolumeSpecName: "kube-api-access-4zz2w") pod "ee09a0fe-af38-40ec-a928-473663d09d0f" (UID: "ee09a0fe-af38-40ec-a928-473663d09d0f"). InnerVolumeSpecName "kube-api-access-4zz2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:04:06.534490 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.534428 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zz2w\" (UniqueName: \"kubernetes.io/projected/ee09a0fe-af38-40ec-a928-473663d09d0f-kube-api-access-4zz2w\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:04:06.534490 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:06.534453 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee09a0fe-af38-40ec-a928-473663d09d0f-webhook-cert\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:04:07.041969 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.041932 2576 generic.go:356] "Generic (PLEG): container finished" podID="ee09a0fe-af38-40ec-a928-473663d09d0f" containerID="43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a" exitCode=0 Jul 29 04:04:07.042468 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.041981 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" Jul 29 04:04:07.042468 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.042047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" event={"ID":"ee09a0fe-af38-40ec-a928-473663d09d0f","Type":"ContainerDied","Data":"43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a"} Jul 29 04:04:07.042468 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.042120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569bbdd56d-n9ts8" event={"ID":"ee09a0fe-af38-40ec-a928-473663d09d0f","Type":"ContainerDied","Data":"022662cdbfb26589b9936e726a7ee4f465ab5ad0c43a3ce340e619e68858be02"} Jul 29 04:04:07.042468 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.042158 2576 scope.go:117] "RemoveContainer" containerID="43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a" Jul 29 04:04:07.052847 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.052628 2576 scope.go:117] "RemoveContainer" containerID="43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a" Jul 29 04:04:07.052965 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:04:07.052941 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a\": container with ID starting with 43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a not found: ID does not exist" containerID="43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a" Jul 29 04:04:07.053023 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.052965 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a"} err="failed to get container status \"43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a\": rpc error: code = NotFound desc = could not find container \"43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a\": container with ID starting with 43f6d5ca9c89dc61f71fa9ed96f2648024070920490cc479098a20c303d0498a not found: ID does not exist" Jul 29 04:04:07.066634 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.066612 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569bbdd56d-n9ts8"] Jul 29 04:04:07.070641 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:07.070622 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-569bbdd56d-n9ts8"] Jul 29 04:04:08.197641 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:08.197610 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee09a0fe-af38-40ec-a928-473663d09d0f" path="/var/lib/kubelet/pods/ee09a0fe-af38-40ec-a928-473663d09d0f/volumes" Jul 29 04:04:26.033250 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.033202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:26.132502 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.132462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:26.818440 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.818397 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn"] Jul 29 04:04:26.818820 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.818808 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee09a0fe-af38-40ec-a928-473663d09d0f" containerName="manager" Jul 29 04:04:26.818867 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.818823 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee09a0fe-af38-40ec-a928-473663d09d0f" containerName="manager" Jul 29 04:04:26.818926 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.818893 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee09a0fe-af38-40ec-a928-473663d09d0f" containerName="manager" Jul 29 04:04:26.823223 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.823205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.825932 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.825905 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 04:04:26.826047 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.825915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\"" Jul 29 04:04:26.834007 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.833984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn"] Jul 29 04:04:26.884838 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.884802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:26.890002 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.889977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a703b6a4-59d0-4e04-ba12-b738620bacce-maas-api-tls\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.890130 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.890054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a703b6a4-59d0-4e04-ba12-b738620bacce-openshift-service-ca\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.890130 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.890078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw56l\" (UniqueName: \"kubernetes.io/projected/a703b6a4-59d0-4e04-ba12-b738620bacce-kube-api-access-jw56l\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.981379 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.981316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:26.990757 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.990732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a703b6a4-59d0-4e04-ba12-b738620bacce-maas-api-tls\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.990919 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.990788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a703b6a4-59d0-4e04-ba12-b738620bacce-openshift-service-ca\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.990919 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.990805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw56l\" (UniqueName: \"kubernetes.io/projected/a703b6a4-59d0-4e04-ba12-b738620bacce-kube-api-access-jw56l\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.991379 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.991356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a703b6a4-59d0-4e04-ba12-b738620bacce-openshift-service-ca\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.993352 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.993330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a703b6a4-59d0-4e04-ba12-b738620bacce-maas-api-tls\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:26.999609 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:26.999585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw56l\" (UniqueName: \"kubernetes.io/projected/a703b6a4-59d0-4e04-ba12-b738620bacce-kube-api-access-jw56l\") pod \"maas-api-74b74d4d58-qs8rn\" (UID: \"a703b6a4-59d0-4e04-ba12-b738620bacce\") " pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:27.134972 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:27.134933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:27.285164 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:27.285139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn"] Jul 29 04:04:27.286910 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:04:27.286865 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda703b6a4_59d0_4e04_ba12_b738620bacce.slice/crio-2c1c6c9ce78378e39000ebaeb0c25131237ebe95908ed19f48abee3f91980f89 WatchSource:0}: Error finding container 2c1c6c9ce78378e39000ebaeb0c25131237ebe95908ed19f48abee3f91980f89: Status 404 returned error can't find the container with id 2c1c6c9ce78378e39000ebaeb0c25131237ebe95908ed19f48abee3f91980f89 Jul 29 04:04:27.298945 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:27.298923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:27.387095 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:27.387014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:28.161102 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:28.161067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" event={"ID":"a703b6a4-59d0-4e04-ba12-b738620bacce","Type":"ContainerStarted","Data":"2c1c6c9ce78378e39000ebaeb0c25131237ebe95908ed19f48abee3f91980f89"} Jul 29 04:04:29.168398 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:29.168303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" event={"ID":"a703b6a4-59d0-4e04-ba12-b738620bacce","Type":"ContainerStarted","Data":"27947575507e4e39fb5baa62ec6d04db51b45633ba5f8222beb73c3e166a0bd3"} Jul 29 04:04:29.168810 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:29.168459 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:29.187049 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:29.187004 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" podStartSLOduration=1.555792826 podStartE2EDuration="3.186990163s" totalImagesPullingTime="1.631197337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:29.185799324 +0000 UTC m=+551.477895686" watchObservedRunningTime="2026-07-29 04:04:29.186990163 +0000 UTC m=+551.479086541" Jul 29 04:04:35.181042 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:35.181013 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74b74d4d58-qs8rn" Jul 29 04:04:37.783310 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:37.783270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:38.008750 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:38.008716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:48.159760 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.159719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:48.320318 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.320290 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx"] Jul 29 04:04:48.323973 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.323957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.326588 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.326557 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 04:04:48.326720 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.326569 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 04:04:48.328076 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.327643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h4cwp\"" Jul 29 04:04:48.328076 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.327867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 29 04:04:48.332691 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.332670 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx"] Jul 29 04:04:48.351552 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.351524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:04:48.380679 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.380645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.380792 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.380732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.380852 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.380828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173252c7-dcf5-420a-8575-3ecaa091e668-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.380915 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.380873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.380954 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.380919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.380988 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.380959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phm8p\" (UniqueName: \"kubernetes.io/projected/173252c7-dcf5-420a-8575-3ecaa091e668-kube-api-access-phm8p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.481693 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.481615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.481693 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.481647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phm8p\" (UniqueName: \"kubernetes.io/projected/173252c7-dcf5-420a-8575-3ecaa091e668-kube-api-access-phm8p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.481693 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.481686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.482000 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.481730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.482000 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.481762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173252c7-dcf5-420a-8575-3ecaa091e668-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.482000 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.481780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.482201 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.482177 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.482260 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.482201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.482302 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.482275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.484457 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.484431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173252c7-dcf5-420a-8575-3ecaa091e668-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.484579 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.484558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173252c7-dcf5-420a-8575-3ecaa091e668-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.489789 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.489759 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phm8p\" (UniqueName: \"kubernetes.io/projected/173252c7-dcf5-420a-8575-3ecaa091e668-kube-api-access-phm8p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx\" (UID: \"173252c7-dcf5-420a-8575-3ecaa091e668\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.637728 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.637684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:48.804674 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:48.804633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx"] Jul 29 04:04:48.806196 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:04:48.806163 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173252c7_dcf5_420a_8575_3ecaa091e668.slice/crio-13521441cec47be1eab0ee11e95384f8989a2cd679cf2769b25a41130f081aab WatchSource:0}: Error finding container 13521441cec47be1eab0ee11e95384f8989a2cd679cf2769b25a41130f081aab: Status 404 returned error can't find the container with id 13521441cec47be1eab0ee11e95384f8989a2cd679cf2769b25a41130f081aab Jul 29 04:04:49.278935 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:49.278893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" event={"ID":"173252c7-dcf5-420a-8575-3ecaa091e668","Type":"ContainerStarted","Data":"13521441cec47be1eab0ee11e95384f8989a2cd679cf2769b25a41130f081aab"} Jul 29 04:04:50.288992 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:50.288915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" event={"ID":"173252c7-dcf5-420a-8575-3ecaa091e668","Type":"ContainerStarted","Data":"5a47cae75385c33b38fbbefc7cad4f32f882b4583bd8b733cc8463d948b92c43"} Jul 29 04:04:50.307675 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:50.307630 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" podStartSLOduration=1.215266795 podStartE2EDuration="2.307595998s" totalImagesPullingTime="1.092329203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:50.305830031 +0000 UTC m=+572.597926393" watchObservedRunningTime="2026-07-29 04:04:50.307595998 +0000 UTC m=+572.599692395" Jul 29 04:04:58.638829 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:58.638792 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:58.639289 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:58.639133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:58.651155 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:58.651133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:04:59.345470 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:04:59.345439 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx" Jul 29 04:05:18.113764 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:18.113736 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:05:18.115482 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:18.115450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:05:20.552147 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.552097 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2"] Jul 29 04:05:20.555809 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.555789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.558494 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.558474 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 29 04:05:20.566175 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.566154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2"] Jul 29 04:05:20.587217 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.587179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:05:20.654246 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.654209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.654483 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.654263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.654483 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.654307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbhv\" (UniqueName: \"kubernetes.io/projected/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-kube-api-access-8zbhv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.654483 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.654325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.654483 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.654354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.654483 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.654377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.754924 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.754876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755120 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.754949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755120 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.755020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbhv\" (UniqueName: \"kubernetes.io/projected/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-kube-api-access-8zbhv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755253 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.755130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755253 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.755180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755253 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.755203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755386 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.755319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755608 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.755568 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.755717 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.755654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.758162 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.758141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.758824 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.758804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.763638 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.763614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbhv\" (UniqueName: \"kubernetes.io/projected/680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd-kube-api-access-8zbhv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-flcm2\" (UID: \"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:20.867186 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:20.867107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:21.013208 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:21.013068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2"] Jul 29 04:05:21.015959 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:05:21.015926 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680f2c4f_1b87_4e9a_af31_15ec2a1cc5fd.slice/crio-fc98fba2003c7ae693845e06799e447b248c04d70b6a7e6a0c562c8305883e66 WatchSource:0}: Error finding container fc98fba2003c7ae693845e06799e447b248c04d70b6a7e6a0c562c8305883e66: Status 404 returned error can't find the container with id fc98fba2003c7ae693845e06799e447b248c04d70b6a7e6a0c562c8305883e66 Jul 29 04:05:21.449480 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:21.449443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" event={"ID":"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd","Type":"ContainerStarted","Data":"5fecc5e8096ac7eae216b025f9bd8596276c8ce0b34ad408c22b5543992546d0"} Jul 29 04:05:21.449675 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:21.449495 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" event={"ID":"680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd","Type":"ContainerStarted","Data":"fc98fba2003c7ae693845e06799e447b248c04d70b6a7e6a0c562c8305883e66"} Jul 29 04:05:21.470948 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:21.470835 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" podStartSLOduration=1.287619686 podStartE2EDuration="1.470819731s" totalImagesPullingTime="183.200045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:05:21.468024008 +0000 UTC m=+603.760120369" watchObservedRunningTime="2026-07-29 04:05:21.470819731 +0000 UTC m=+603.762916093" Jul 29 04:05:30.867945 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:30.867912 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:30.868353 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:30.867981 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:30.880335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:30.880313 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:31.515220 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:31.515194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-flcm2" Jul 29 04:05:36.063388 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:36.063350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:05:52.068242 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:05:52.068210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:06:05.228189 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.228158 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz"] Jul 29 04:06:05.231735 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.231720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.234366 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.234349 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 29 04:06:05.241021 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.240998 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz"] Jul 29 04:06:05.335454 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.335425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:06:05.353495 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.353472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.353645 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.353518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f696de-2366-4b75-8787-d7c3114a8a13-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.353645 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.353539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhj5\" (UniqueName: \"kubernetes.io/projected/e0f696de-2366-4b75-8787-d7c3114a8a13-kube-api-access-xhhj5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.353645 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.353611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.353825 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.353726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.353825 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.353784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454214 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454363 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454363 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f696de-2366-4b75-8787-d7c3114a8a13-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454363 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhj5\" (UniqueName: \"kubernetes.io/projected/e0f696de-2366-4b75-8787-d7c3114a8a13-kube-api-access-xhhj5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454363 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454593 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454669 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454731 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.454773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.454750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.456763 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.456742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f696de-2366-4b75-8787-d7c3114a8a13-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.456935 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.456918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f696de-2366-4b75-8787-d7c3114a8a13-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.462178 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.462154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhj5\" (UniqueName: \"kubernetes.io/projected/e0f696de-2366-4b75-8787-d7c3114a8a13-kube-api-access-xhhj5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4ztxz\" (UID: \"e0f696de-2366-4b75-8787-d7c3114a8a13\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.543577 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.543507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:05.683527 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.683494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz"] Jul 29 04:06:05.686712 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.686689 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:06:05.693145 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:05.693120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" event={"ID":"e0f696de-2366-4b75-8787-d7c3114a8a13","Type":"ContainerStarted","Data":"f46e6925f84b8ddcddb8b92b12a1a765729393252be29be51f9126f9426da558"} Jul 29 04:06:06.699607 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:06.699567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" event={"ID":"e0f696de-2366-4b75-8787-d7c3114a8a13","Type":"ContainerStarted","Data":"c87818efd6a8db991a9d95a528a6e69f7c7c3c9b1c350daa39b6b8aec20478b5"} Jul 29 04:06:06.722047 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:06.721993 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" podStartSLOduration=1.551056387 podStartE2EDuration="1.721979442s" totalImagesPullingTime="170.923055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:06:06.719715748 +0000 UTC m=+649.011812112" watchObservedRunningTime="2026-07-29 04:06:06.721979442 +0000 UTC m=+649.014075804" Jul 29 04:06:15.133447 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:15.133409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:06:15.544636 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:15.544557 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:15.544636 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:15.544623 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:15.557012 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:15.556989 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:06:15.757971 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:06:15.757944 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4ztxz" Jul 29 04:07:06.495617 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:06.495549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:07:06.587188 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:06.587146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:07:07.634006 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:07.633974 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:07:07.736443 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:07.736414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:07:16.794237 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:16.794156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:07:53.056716 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.056679 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8cdf94955-j6v9p"] Jul 29 04:07:53.057336 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.056915 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8cdf94955-j6v9p" podUID="13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" containerName="manager" containerID="cri-o://aabd305fe45a38aa071792a220333e90df3d48798c492a8d2011a6dfc5faf47f" gracePeriod=30 Jul 29 04:07:53.282674 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.282639 2576 generic.go:356] "Generic (PLEG): container finished" podID="13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" containerID="aabd305fe45a38aa071792a220333e90df3d48798c492a8d2011a6dfc5faf47f" exitCode=0 Jul 29 04:07:53.282877 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.282698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cdf94955-j6v9p" event={"ID":"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e","Type":"ContainerDied","Data":"aabd305fe45a38aa071792a220333e90df3d48798c492a8d2011a6dfc5faf47f"} Jul 29 04:07:53.314591 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.314528 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:07:53.460809 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.460778 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhdhh\" (UniqueName: \"kubernetes.io/projected/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-kube-api-access-bhdhh\") pod \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " Jul 29 04:07:53.460809 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.460816 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-webhook-cert\") pod \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\" (UID: \"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e\") " Jul 29 04:07:53.463088 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.463054 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" (UID: "13f9f794-6bca-4944-bdc2-a69e9d7d5c2e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:07:53.463215 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.463106 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-kube-api-access-bhdhh" (OuterVolumeSpecName: "kube-api-access-bhdhh") pod "13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" (UID: "13f9f794-6bca-4944-bdc2-a69e9d7d5c2e"). InnerVolumeSpecName "kube-api-access-bhdhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:07:53.562348 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.562310 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdhh\" (UniqueName: \"kubernetes.io/projected/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-kube-api-access-bhdhh\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:07:53.562348 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:53.562341 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e-webhook-cert\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:07:54.289328 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:54.289291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cdf94955-j6v9p" event={"ID":"13f9f794-6bca-4944-bdc2-a69e9d7d5c2e","Type":"ContainerDied","Data":"d9c660136ca53dcd65715fb139c641b585b33228ddc53acb0e768df5b3b17e31"} Jul 29 04:07:54.289726 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:54.289351 2576 scope.go:117] "RemoveContainer" containerID="aabd305fe45a38aa071792a220333e90df3d48798c492a8d2011a6dfc5faf47f" Jul 29 04:07:54.289726 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:54.289305 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cdf94955-j6v9p" Jul 29 04:07:54.308093 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:54.308072 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8cdf94955-j6v9p"] Jul 29 04:07:54.311821 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:54.311797 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8cdf94955-j6v9p"] Jul 29 04:07:56.197178 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:07:56.197147 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" path="/var/lib/kubelet/pods/13f9f794-6bca-4944-bdc2-a69e9d7d5c2e/volumes" Jul 29 04:08:29.995819 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:08:29.995767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:08:31.164188 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:08:31.164157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:08:31.218686 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:08:31.218661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:08:33.968908 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:08:33.968854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:09:13.870429 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:09:13.870394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:09:15.090930 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:09:15.090898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:17.813987 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:17.813873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:17.911219 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:17.911187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:18.020909 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:18.020853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:18.149862 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:18.149829 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:10:18.151064 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:18.151039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:10:23.952942 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:23.952905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:24.053414 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:24.053382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:24.151421 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:24.151374 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:24.294155 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:24.294067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:29.972383 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:29.972349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:30.072636 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:30.072606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:10:52.752279 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:10:52.752239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:13:30.256356 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:13:30.256278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:14:10.849443 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:14:10.849410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:15:00.142875 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.142832 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp"] Jul 29 04:15:00.143697 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.143673 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" containerName="manager" Jul 29 04:15:00.143867 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.143854 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" containerName="manager" Jul 29 04:15:00.144217 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.144201 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f9f794-6bca-4944-bdc2-a69e9d7d5c2e" containerName="manager" Jul 29 04:15:00.148289 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.148269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp" Jul 29 04:15:00.151121 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.151103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\"" Jul 29 04:15:00.158319 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.158300 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp"] Jul 29 04:15:00.158841 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.158812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp" Jul 29 04:15:00.299928 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.299903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp"] Jul 29 04:15:00.301663 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:15:00.301629 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239d5efe_ea70_4477_9c15_dee88469b3af.slice/crio-368bcae251fd4b5fbd2d367499bac7d688f763d7bdf7149b0e7a0712d08185c5 WatchSource:0}: Error finding container 368bcae251fd4b5fbd2d367499bac7d688f763d7bdf7149b0e7a0712d08185c5: Status 404 returned error can't find the container with id 368bcae251fd4b5fbd2d367499bac7d688f763d7bdf7149b0e7a0712d08185c5 Jul 29 04:15:00.303669 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.303651 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:15:00.569701 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:00.569616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp" event={"ID":"239d5efe-ea70-4477-9c15-dee88469b3af","Type":"ContainerStarted","Data":"368bcae251fd4b5fbd2d367499bac7d688f763d7bdf7149b0e7a0712d08185c5"} Jul 29 04:15:03.590467 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:03.590440 2576 generic.go:356] "Generic (PLEG): container finished" podID="239d5efe-ea70-4477-9c15-dee88469b3af" containerID="0b07c0a50a257d7cff931becec4bc2f1c8e6da15e1718c7ca70f37bb54555f00" exitCode=0 Jul 29 04:15:03.590811 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:03.590518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp" event={"ID":"239d5efe-ea70-4477-9c15-dee88469b3af","Type":"ContainerDied","Data":"0b07c0a50a257d7cff931becec4bc2f1c8e6da15e1718c7ca70f37bb54555f00"} Jul 29 04:15:04.734107 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:04.734083 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp" Jul 29 04:15:05.601588 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:05.601555 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp" Jul 29 04:15:05.601773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:05.601606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp" event={"ID":"239d5efe-ea70-4477-9c15-dee88469b3af","Type":"ContainerDied","Data":"368bcae251fd4b5fbd2d367499bac7d688f763d7bdf7149b0e7a0712d08185c5"} Jul 29 04:15:05.601773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:05.601652 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368bcae251fd4b5fbd2d367499bac7d688f763d7bdf7149b0e7a0712d08185c5" Jul 29 04:15:14.427751 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:14.427701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:15:18.185354 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:18.185320 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:15:18.188139 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:15:18.188117 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:20:18.220873 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:20:18.220845 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:20:18.227498 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:20:18.227478 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:20:35.940785 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:20:35.940751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:20:37.365071 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:20:37.365033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:20:56.770693 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:20:56.770657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:20:59.701994 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:20:59.701961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:21:05.965334 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:21:05.965302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:21:52.642862 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:21:52.642826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:21:53.802856 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:21:53.802824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:22:16.283519 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:22:16.283449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:25:18.266249 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:25:18.266219 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:25:18.270337 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:25:18.270313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:25:54.662309 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:25:54.662263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:29.473139 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:29.473104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:30.999059 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:30.999026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:34.030529 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:34.030492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:34.134979 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:34.134947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:34.460206 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:34.460164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:39.779540 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:39.779448 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:39.879576 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:39.879538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:39.993665 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:39.993633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:40.094468 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:40.094382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:40.213474 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:40.213439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:40.501766 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:40.501740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:40.615476 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:40.615445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:43.226961 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.226922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:43.322876 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.322839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:43.323106 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.323086 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq"] Jul 29 04:26:43.323501 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.323483 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239d5efe-ea70-4477-9c15-dee88469b3af" containerName="cleanup" Jul 29 04:26:43.323501 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.323498 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="239d5efe-ea70-4477-9c15-dee88469b3af" containerName="cleanup" Jul 29 04:26:43.323667 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.323564 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="239d5efe-ea70-4477-9c15-dee88469b3af" containerName="cleanup" Jul 29 04:26:43.326648 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.326631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.329303 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.329282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dd29cdf8-serving-cert\"" Jul 29 04:26:43.337449 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.337427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq"] Jul 29 04:26:43.390498 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.390471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:43.447835 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.447804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77z2\" (UniqueName: \"kubernetes.io/projected/a2b5b47c-11af-4c81-8953-00871e821ba4-kube-api-access-k77z2\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.448015 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.447898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b5b47c-11af-4c81-8953-00871e821ba4-openshift-service-ca\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.448015 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.448001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.549037 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.548952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.549037 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.549002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77z2\" (UniqueName: \"kubernetes.io/projected/a2b5b47c-11af-4c81-8953-00871e821ba4-kube-api-access-k77z2\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.549224 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.549067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b5b47c-11af-4c81-8953-00871e821ba4-openshift-service-ca\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.549224 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:26:43.549100 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-serving-cert: secret "maas-api-e2e-ait-dd29cdf8-serving-cert" not found Jul 29 04:26:43.549224 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:26:43.549166 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls podName:a2b5b47c-11af-4c81-8953-00871e821ba4 nodeName:}" failed. No retries permitted until 2026-07-29 04:26:44.04914994 +0000 UTC m=+1886.341246280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls") pod "maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" (UID: "a2b5b47c-11af-4c81-8953-00871e821ba4") : secret "maas-api-e2e-ait-dd29cdf8-serving-cert" not found Jul 29 04:26:43.549701 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.549683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b5b47c-11af-4c81-8953-00871e821ba4-openshift-service-ca\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.558730 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.558698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77z2\" (UniqueName: \"kubernetes.io/projected/a2b5b47c-11af-4c81-8953-00871e821ba4-kube-api-access-k77z2\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:43.766460 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.766422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:43.983245 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:43.983217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:44.054807 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:44.054771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:44.057456 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:44.057432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls\") pod \"maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:44.079490 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:44.079462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:26:44.238535 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:44.238457 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:44.378773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:44.378748 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq"] Jul 29 04:26:44.380472 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:26:44.380441 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b5b47c_11af_4c81_8953_00871e821ba4.slice/crio-4954e35007461d979fbc1f84e3d293a57c3298b0263c8635fe18d20a44251540 WatchSource:0}: Error finding container 4954e35007461d979fbc1f84e3d293a57c3298b0263c8635fe18d20a44251540: Status 404 returned error can't find the container with id 4954e35007461d979fbc1f84e3d293a57c3298b0263c8635fe18d20a44251540 Jul 29 04:26:44.381780 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:44.381760 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:26:44.391920 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:44.391893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" event={"ID":"a2b5b47c-11af-4c81-8953-00871e821ba4","Type":"ContainerStarted","Data":"4954e35007461d979fbc1f84e3d293a57c3298b0263c8635fe18d20a44251540"} Jul 29 04:26:45.398319 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:45.398284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" event={"ID":"a2b5b47c-11af-4c81-8953-00871e821ba4","Type":"ContainerStarted","Data":"99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3"} Jul 29 04:26:45.398714 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:45.398353 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:26:45.417203 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:45.417151 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" podStartSLOduration=1.9538820270000001 podStartE2EDuration="2.417134935s" totalImagesPullingTime="463.252908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:26:45.414341912 +0000 UTC m=+1887.706438277" watchObservedRunningTime="2026-07-29 04:26:45.417134935 +0000 UTC m=+1887.709231297" Jul 29 04:26:51.409544 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:26:51.409515 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:27:23.907477 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:23.907442 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq"] Jul 29 04:27:23.909797 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:23.908190 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" podUID="a2b5b47c-11af-4c81-8953-00871e821ba4" containerName="maas-api" containerID="cri-o://99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3" gracePeriod=30 Jul 29 04:27:23.936008 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:23.935981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.000342 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.000296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.058978 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.058946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.116368 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.116334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.171411 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.171359 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:27:24.246741 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.246713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.274384 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.274357 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b5b47c-11af-4c81-8953-00871e821ba4-openshift-service-ca\") pod \"a2b5b47c-11af-4c81-8953-00871e821ba4\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " Jul 29 04:27:24.274553 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.274424 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k77z2\" (UniqueName: \"kubernetes.io/projected/a2b5b47c-11af-4c81-8953-00871e821ba4-kube-api-access-k77z2\") pod \"a2b5b47c-11af-4c81-8953-00871e821ba4\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " Jul 29 04:27:24.274553 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.274470 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls\") pod \"a2b5b47c-11af-4c81-8953-00871e821ba4\" (UID: \"a2b5b47c-11af-4c81-8953-00871e821ba4\") " Jul 29 04:27:24.274774 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.274747 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b5b47c-11af-4c81-8953-00871e821ba4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2b5b47c-11af-4c81-8953-00871e821ba4" (UID: "a2b5b47c-11af-4c81-8953-00871e821ba4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:27:24.277377 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.277346 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2b5b47c-11af-4c81-8953-00871e821ba4" (UID: "a2b5b47c-11af-4c81-8953-00871e821ba4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:27:24.277568 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.277544 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b5b47c-11af-4c81-8953-00871e821ba4-kube-api-access-k77z2" (OuterVolumeSpecName: "kube-api-access-k77z2") pod "a2b5b47c-11af-4c81-8953-00871e821ba4" (UID: "a2b5b47c-11af-4c81-8953-00871e821ba4"). InnerVolumeSpecName "kube-api-access-k77z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:27:24.375157 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.375123 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77z2\" (UniqueName: \"kubernetes.io/projected/a2b5b47c-11af-4c81-8953-00871e821ba4-kube-api-access-k77z2\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:27:24.375157 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.375146 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b5b47c-11af-4c81-8953-00871e821ba4-maas-api-tls\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:27:24.375157 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.375156 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b5b47c-11af-4c81-8953-00871e821ba4-openshift-service-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:27:24.470634 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.470552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.606864 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.606833 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2b5b47c-11af-4c81-8953-00871e821ba4" containerID="99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3" exitCode=0 Jul 29 04:27:24.607037 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.606908 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" Jul 29 04:27:24.607037 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.606926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" event={"ID":"a2b5b47c-11af-4c81-8953-00871e821ba4","Type":"ContainerDied","Data":"99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3"} Jul 29 04:27:24.607037 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.606986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq" event={"ID":"a2b5b47c-11af-4c81-8953-00871e821ba4","Type":"ContainerDied","Data":"4954e35007461d979fbc1f84e3d293a57c3298b0263c8635fe18d20a44251540"} Jul 29 04:27:24.607037 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.607016 2576 scope.go:117] "RemoveContainer" containerID="99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3" Jul 29 04:27:24.617181 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.617161 2576 scope.go:117] "RemoveContainer" containerID="99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3" Jul 29 04:27:24.617458 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:27:24.617436 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3\": container with ID starting with 99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3 not found: ID does not exist" containerID="99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3" Jul 29 04:27:24.617522 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.617471 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3"} err="failed to get container status \"99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3\": rpc error: code = NotFound desc = could not find container \"99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3\": container with ID starting with 99cde58433a7bf072bb7bc28acfe54dbec8adc97f07806888cfa442655546bd3 not found: ID does not exist" Jul 29 04:27:24.627963 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.627940 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq"] Jul 29 04:27:24.633445 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.633421 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd29cdf8-5fbcdbc64f-jftcq"] Jul 29 04:27:24.735402 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.735307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.835228 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.835200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:24.938950 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:24.938914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:25.326931 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:25.326864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:26.028851 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:26.028810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:26.197961 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:26.197924 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b5b47c-11af-4c81-8953-00871e821ba4" path="/var/lib/kubelet/pods/a2b5b47c-11af-4c81-8953-00871e821ba4/volumes" Jul 29 04:27:27.535907 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:27.535850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:27.640542 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:27.640510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:27.875759 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:27.875727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:28.131486 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:28.131412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:28.230397 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:28.230368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:52.534387 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.534353 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt"] Jul 29 04:27:52.534748 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.534739 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b5b47c-11af-4c81-8953-00871e821ba4" containerName="maas-api" Jul 29 04:27:52.534791 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.534750 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b5b47c-11af-4c81-8953-00871e821ba4" containerName="maas-api" Jul 29 04:27:52.534822 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.534817 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b5b47c-11af-4c81-8953-00871e821ba4" containerName="maas-api" Jul 29 04:27:52.537771 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.537755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" Jul 29 04:27:52.540172 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.540143 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\"" Jul 29 04:27:52.545130 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.545096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt"] Jul 29 04:27:52.625125 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.625104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be019ee1-c186-4b60-894b-48eeb8a9b286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-74690011-d76mt\" (UID: \"be019ee1-c186-4b60-894b-48eeb8a9b286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" Jul 29 04:27:52.726486 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.726447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be019ee1-c186-4b60-894b-48eeb8a9b286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-74690011-d76mt\" (UID: \"be019ee1-c186-4b60-894b-48eeb8a9b286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" Jul 29 04:27:52.727165 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.727148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be019ee1-c186-4b60-894b-48eeb8a9b286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-74690011-d76mt\" (UID: \"be019ee1-c186-4b60-894b-48eeb8a9b286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" Jul 29 04:27:52.849898 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.849801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" Jul 29 04:27:52.986342 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:52.986317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt"] Jul 29 04:27:52.987500 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:27:52.987477 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe019ee1_c186_4b60_894b_48eeb8a9b286.slice/crio-0fc317bf11358db10bbe89c90f7066473544d0506e7037e25aae9e2345adeb99 WatchSource:0}: Error finding container 0fc317bf11358db10bbe89c90f7066473544d0506e7037e25aae9e2345adeb99: Status 404 returned error can't find the container with id 0fc317bf11358db10bbe89c90f7066473544d0506e7037e25aae9e2345adeb99 Jul 29 04:27:53.767119 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:53.767086 2576 generic.go:356] "Generic (PLEG): container finished" podID="be019ee1-c186-4b60-894b-48eeb8a9b286" containerID="d3e74a35e35307b445386e3a22c04fa9a788a30d46880629c3ad36224a4bd544" exitCode=0 Jul 29 04:27:53.767436 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:53.767178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" event={"ID":"be019ee1-c186-4b60-894b-48eeb8a9b286","Type":"ContainerDied","Data":"d3e74a35e35307b445386e3a22c04fa9a788a30d46880629c3ad36224a4bd544"} Jul 29 04:27:53.767436 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:53.767268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" event={"ID":"be019ee1-c186-4b60-894b-48eeb8a9b286","Type":"ContainerStarted","Data":"0fc317bf11358db10bbe89c90f7066473544d0506e7037e25aae9e2345adeb99"} Jul 29 04:27:54.912467 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:54.912445 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" Jul 29 04:27:55.044315 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:55.044232 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be019ee1-c186-4b60-894b-48eeb8a9b286-maas-api-service-ca\") pod \"be019ee1-c186-4b60-894b-48eeb8a9b286\" (UID: \"be019ee1-c186-4b60-894b-48eeb8a9b286\") " Jul 29 04:27:55.044486 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:55.044466 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be019ee1-c186-4b60-894b-48eeb8a9b286-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be019ee1-c186-4b60-894b-48eeb8a9b286" (UID: "be019ee1-c186-4b60-894b-48eeb8a9b286"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:27:55.145831 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:55.145796 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be019ee1-c186-4b60-894b-48eeb8a9b286-maas-api-service-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:27:55.779601 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:55.779562 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" Jul 29 04:27:55.779803 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:55.779592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt" event={"ID":"be019ee1-c186-4b60-894b-48eeb8a9b286","Type":"ContainerDied","Data":"0fc317bf11358db10bbe89c90f7066473544d0506e7037e25aae9e2345adeb99"} Jul 29 04:27:55.779803 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:55.779645 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc317bf11358db10bbe89c90f7066473544d0506e7037e25aae9e2345adeb99" Jul 29 04:27:59.050130 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.050085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:59.150323 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.150293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:59.201992 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.201960 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt"] Jul 29 04:27:59.204351 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.204331 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-74690011-d76mt"] Jul 29 04:27:59.207149 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.207122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:59.343236 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.343158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:59.564575 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.564538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:27:59.809200 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:27:59.809162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:00.106162 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:00.106074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:00.197022 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:00.196979 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be019ee1-c186-4b60-894b-48eeb8a9b286" path="/var/lib/kubelet/pods/be019ee1-c186-4b60-894b-48eeb8a9b286/volumes" Jul 29 04:28:00.204322 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:00.204297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:02.587954 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:02.587922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:02.666718 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:02.666681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:02.742534 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:02.742500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:02.890563 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:02.890536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:03.293461 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:03.293378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:03.392672 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:03.392634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:28.387800 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.387763 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g"] Jul 29 04:28:28.388290 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.388168 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be019ee1-c186-4b60-894b-48eeb8a9b286" containerName="revoke-keys" Jul 29 04:28:28.388290 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.388180 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="be019ee1-c186-4b60-894b-48eeb8a9b286" containerName="revoke-keys" Jul 29 04:28:28.388290 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.388259 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="be019ee1-c186-4b60-894b-48eeb8a9b286" containerName="revoke-keys" Jul 29 04:28:28.391386 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.391371 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" Jul 29 04:28:28.393816 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.393795 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\"" Jul 29 04:28:28.397930 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.397906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g"] Jul 29 04:28:28.430564 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.430528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aa5625-dfa3-4bca-8b8b-55a31b585269-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g\" (UID: \"a3aa5625-dfa3-4bca-8b8b-55a31b585269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" Jul 29 04:28:28.531284 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.531253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aa5625-dfa3-4bca-8b8b-55a31b585269-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g\" (UID: \"a3aa5625-dfa3-4bca-8b8b-55a31b585269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" Jul 29 04:28:28.531866 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.531845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aa5625-dfa3-4bca-8b8b-55a31b585269-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g\" (UID: \"a3aa5625-dfa3-4bca-8b8b-55a31b585269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" Jul 29 04:28:28.703123 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.703025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" Jul 29 04:28:28.838382 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.838353 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g"] Jul 29 04:28:28.840081 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:28:28.840056 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3aa5625_dfa3_4bca_8b8b_55a31b585269.slice/crio-cc0ae40523d338599ec40b21e232a6e4695d8808d598accc3b682c9f3b3de54e WatchSource:0}: Error finding container cc0ae40523d338599ec40b21e232a6e4695d8808d598accc3b682c9f3b3de54e: Status 404 returned error can't find the container with id cc0ae40523d338599ec40b21e232a6e4695d8808d598accc3b682c9f3b3de54e Jul 29 04:28:28.971834 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:28.971751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" event={"ID":"a3aa5625-dfa3-4bca-8b8b-55a31b585269","Type":"ContainerStarted","Data":"cc0ae40523d338599ec40b21e232a6e4695d8808d598accc3b682c9f3b3de54e"} Jul 29 04:28:29.978606 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:29.978568 2576 generic.go:356] "Generic (PLEG): container finished" podID="a3aa5625-dfa3-4bca-8b8b-55a31b585269" containerID="008cf6bdaae93c633bf2374a9d21ef9c9615ca90c56130b24b94340a5973fa56" exitCode=0 Jul 29 04:28:29.979013 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:29.978605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" event={"ID":"a3aa5625-dfa3-4bca-8b8b-55a31b585269","Type":"ContainerDied","Data":"008cf6bdaae93c633bf2374a9d21ef9c9615ca90c56130b24b94340a5973fa56"} Jul 29 04:28:31.121142 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:31.121117 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" Jul 29 04:28:31.153493 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:31.153471 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aa5625-dfa3-4bca-8b8b-55a31b585269-maas-api-service-ca\") pod \"a3aa5625-dfa3-4bca-8b8b-55a31b585269\" (UID: \"a3aa5625-dfa3-4bca-8b8b-55a31b585269\") " Jul 29 04:28:31.153773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:31.153753 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3aa5625-dfa3-4bca-8b8b-55a31b585269-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3aa5625-dfa3-4bca-8b8b-55a31b585269" (UID: "a3aa5625-dfa3-4bca-8b8b-55a31b585269"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:28:31.254212 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:31.254154 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aa5625-dfa3-4bca-8b8b-55a31b585269-maas-api-service-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:28:31.991130 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:31.991100 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" Jul 29 04:28:31.991298 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:31.991128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g" event={"ID":"a3aa5625-dfa3-4bca-8b8b-55a31b585269","Type":"ContainerDied","Data":"cc0ae40523d338599ec40b21e232a6e4695d8808d598accc3b682c9f3b3de54e"} Jul 29 04:28:31.991298 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:31.991175 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc0ae40523d338599ec40b21e232a6e4695d8808d598accc3b682c9f3b3de54e" Jul 29 04:28:33.339756 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:33.339722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:33.397711 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:33.397669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:33.459118 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:33.459091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:33.484120 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:33.484083 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g"] Jul 29 04:28:33.486423 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:33.486401 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dcdbfe9-59z7g"] Jul 29 04:28:33.516194 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:33.516169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:34.196578 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:34.196547 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3aa5625-dfa3-4bca-8b8b-55a31b585269" path="/var/lib/kubelet/pods/a3aa5625-dfa3-4bca-8b8b-55a31b585269/volumes" Jul 29 04:28:34.644991 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:34.644960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:34.853635 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:34.853602 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:35.066582 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:35.066488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:35.366051 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:35.365974 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:35.740008 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:35.739973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:37.762101 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:37.762069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:37.838246 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:37.838214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:37.899472 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:37.899441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:38.204721 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:38.204687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:38.545942 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:38.545841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:28:38.646977 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:28:38.646941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:02.641763 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.641730 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct"] Jul 29 04:29:02.642293 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.642164 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3aa5625-dfa3-4bca-8b8b-55a31b585269" containerName="revoke-keys" Jul 29 04:29:02.642293 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.642179 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3aa5625-dfa3-4bca-8b8b-55a31b585269" containerName="revoke-keys" Jul 29 04:29:02.642293 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.642240 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3aa5625-dfa3-4bca-8b8b-55a31b585269" containerName="revoke-keys" Jul 29 04:29:02.645433 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.645417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" Jul 29 04:29:02.648232 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.648199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\"" Jul 29 04:29:02.651535 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.651505 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct"] Jul 29 04:29:02.827802 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.827772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3524547f-2c7a-47ec-bebd-30bf3a7072b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a77d4166-l88ct\" (UID: \"3524547f-2c7a-47ec-bebd-30bf3a7072b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" Jul 29 04:29:02.928853 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.928765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3524547f-2c7a-47ec-bebd-30bf3a7072b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a77d4166-l88ct\" (UID: \"3524547f-2c7a-47ec-bebd-30bf3a7072b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" Jul 29 04:29:02.929438 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.929418 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3524547f-2c7a-47ec-bebd-30bf3a7072b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a77d4166-l88ct\" (UID: \"3524547f-2c7a-47ec-bebd-30bf3a7072b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" Jul 29 04:29:02.958441 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:02.958408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" Jul 29 04:29:03.097714 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:03.097684 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct"] Jul 29 04:29:03.098942 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:29:03.098917 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3524547f_2c7a_47ec_bebd_30bf3a7072b8.slice/crio-4ef726753f9850952d174d2f6493ba5ad2d5a859db8e10e90d8c7ea3cbac75d2 WatchSource:0}: Error finding container 4ef726753f9850952d174d2f6493ba5ad2d5a859db8e10e90d8c7ea3cbac75d2: Status 404 returned error can't find the container with id 4ef726753f9850952d174d2f6493ba5ad2d5a859db8e10e90d8c7ea3cbac75d2 Jul 29 04:29:03.166065 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:03.166027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" event={"ID":"3524547f-2c7a-47ec-bebd-30bf3a7072b8","Type":"ContainerStarted","Data":"4ef726753f9850952d174d2f6493ba5ad2d5a859db8e10e90d8c7ea3cbac75d2"} Jul 29 04:29:04.173800 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:04.173772 2576 generic.go:356] "Generic (PLEG): container finished" podID="3524547f-2c7a-47ec-bebd-30bf3a7072b8" containerID="19d2d3ef69be39932549610861c8e77a8d11face4fe218253d120758b6ff27f0" exitCode=0 Jul 29 04:29:04.174161 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:04.173855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" event={"ID":"3524547f-2c7a-47ec-bebd-30bf3a7072b8","Type":"ContainerDied","Data":"19d2d3ef69be39932549610861c8e77a8d11face4fe218253d120758b6ff27f0"} Jul 29 04:29:05.311935 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:05.311913 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" Jul 29 04:29:05.450394 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:05.450328 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3524547f-2c7a-47ec-bebd-30bf3a7072b8-maas-api-service-ca\") pod \"3524547f-2c7a-47ec-bebd-30bf3a7072b8\" (UID: \"3524547f-2c7a-47ec-bebd-30bf3a7072b8\") " Jul 29 04:29:05.450659 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:05.450637 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3524547f-2c7a-47ec-bebd-30bf3a7072b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3524547f-2c7a-47ec-bebd-30bf3a7072b8" (UID: "3524547f-2c7a-47ec-bebd-30bf3a7072b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:29:05.550866 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:05.550832 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3524547f-2c7a-47ec-bebd-30bf3a7072b8-maas-api-service-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:29:06.185993 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:06.185962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" event={"ID":"3524547f-2c7a-47ec-bebd-30bf3a7072b8","Type":"ContainerDied","Data":"4ef726753f9850952d174d2f6493ba5ad2d5a859db8e10e90d8c7ea3cbac75d2"} Jul 29 04:29:06.185993 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:06.185993 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct" Jul 29 04:29:06.186207 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:06.186016 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef726753f9850952d174d2f6493ba5ad2d5a859db8e10e90d8c7ea3cbac75d2" Jul 29 04:29:07.526749 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:07.526706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:08.402802 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.402769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:08.472861 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.472813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:08.557903 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.557849 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct"] Jul 29 04:29:08.561068 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.561038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:08.562140 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.562120 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a77d4166-l88ct"] Jul 29 04:29:08.614990 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.614946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:08.715921 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.715788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:08.923265 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:08.923231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:09.384069 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:09.384031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:09.486789 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:09.486759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:09.587299 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:09.587270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:10.197713 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:10.197685 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3524547f-2c7a-47ec-bebd-30bf3a7072b8" path="/var/lib/kubelet/pods/3524547f-2c7a-47ec-bebd-30bf3a7072b8/volumes" Jul 29 04:29:31.494515 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:31.494479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:37.599410 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:37.599370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:37.703946 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:37.703909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:37.797267 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:37.797231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:37.896999 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:37.896959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:29:38.328867 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:29:38.328787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:04.291328 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:04.291289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:10.830360 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:10.830319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:10.937916 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:10.937826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:11.032268 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:11.032236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:18.302574 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:18.302545 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:30:18.310151 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:18.310124 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:30:26.926792 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:26.926761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:32.618711 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:32.618663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:32.725281 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:32.725246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:32.819946 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:32.819904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:35.289268 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:35.289225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:35.357195 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:35.357158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:43.319958 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:43.319906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:30:54.300700 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:30:54.300667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:31:00.390206 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:31:00.390169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:31:01.463479 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:31:01.463439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:31:36.981284 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:31:36.981192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:02.495800 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:02.495762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:09.061237 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:09.061205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:09.160537 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:09.160502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:09.817899 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:09.817854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:10.115045 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:10.114949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:10.565085 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:10.565049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:12.810174 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:12.810134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:12.898780 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:12.898749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:12.999199 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:12.999174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:13.630076 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:13.630025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:13.711629 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:13.711597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:37.818184 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:37.818143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:38.458410 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:38.458380 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:38.562828 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:38.562790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:38.894517 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:38.894462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:44.850920 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:44.850853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:44.919803 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:44.919773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:55.823077 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:55.823044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:55.940481 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:55.940443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:56.027183 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:56.027139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:56.096771 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:56.096703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:33:56.463956 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:33:56.463924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:02.097825 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:02.097790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:02.208663 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:02.208623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:02.391013 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:02.390986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:02.643848 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:02.643759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:02.767489 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:02.767455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:02.866000 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:02.865966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:02.993050 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:02.992967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:06.320990 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:06.320957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:08.647362 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:08.647276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:08.754063 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:08.754027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:08.848849 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:08.848819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:09.007165 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:09.007076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:15.032170 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:15.032136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:15.095628 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:15.095590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:34:18.398984 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:18.398955 2576 scope.go:117] "RemoveContainer" containerID="d3e74a35e35307b445386e3a22c04fa9a788a30d46880629c3ad36224a4bd544" Jul 29 04:34:28.635491 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:34:28.635431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:35:18.337767 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:35:18.337736 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:35:18.347909 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:35:18.347856 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:35:18.431183 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:35:18.431152 2576 scope.go:117] "RemoveContainer" containerID="19d2d3ef69be39932549610861c8e77a8d11face4fe218253d120758b6ff27f0" Jul 29 04:35:18.445220 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:35:18.445196 2576 scope.go:117] "RemoveContainer" containerID="008cf6bdaae93c633bf2374a9d21ef9c9615ca90c56130b24b94340a5973fa56" Jul 29 04:36:17.235104 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:17.235069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:19.648665 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:19.648634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:20.424963 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:20.424922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:20.517775 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:20.517736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:20.595968 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:20.595929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:21.103969 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:21.103834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:21.320169 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:21.320135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:21.413288 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:21.413262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:45.366138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:45.366095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:36:48.768707 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:36:48.768675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:37:27.894635 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:37:27.894599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:37:35.158224 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:37:35.158181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:37:35.255288 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:37:35.255254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:37:35.340099 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:37:35.340071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:37:35.409036 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:37:35.408958 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:37:38.481830 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:37:38.481794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:37:43.500856 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:37:43.500823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:20.487284 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:20.487255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:20.640655 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:20.640600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:26.460074 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:26.460042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:49.539582 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:49.539503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:49.639970 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:49.639939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:49.823365 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:49.823288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:50.047669 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:50.047638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:38:55.349186 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:38:55.349154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:11.627834 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:11.627795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:11.730133 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:11.730102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:11.826691 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:11.826636 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:22.780689 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:22.780643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:22.877348 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:22.877293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:22.945147 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:22.945105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:24.310960 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:24.310917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:30.407433 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:30.407399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:30.508834 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:30.508808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:31.109531 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:31.109492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:37.435747 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:37.435719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:37.568431 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:37.568400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:37.704535 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:37.704442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:48.155454 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:48.155422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:39:58.727432 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:39:58.727400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:18.375659 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:18.375589 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:40:18.385000 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:18.384981 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:40:19.708991 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:19.708954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:26.252815 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:26.252781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:26.349838 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:26.349808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:26.604995 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:26.604916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:26.953703 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:26.953671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:27.194924 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:27.194875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:29.457859 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:29.457830 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:29.554676 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:29.554646 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:29.711528 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:29.711444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:30.258433 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:30.258396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:30.361139 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:30.361095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:30.445052 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:30.445019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:35.710107 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:35.710072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:35.770709 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:35.770659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:54.460487 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:54.460456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:54.830953 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:54.830850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:55.105302 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:55.105226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:55.425549 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:55.425520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:57.579752 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.579720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:57.678555 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.678527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:57.826094 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.826066 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5"] Jul 29 04:40:57.826592 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.826569 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3524547f-2c7a-47ec-bebd-30bf3a7072b8" containerName="revoke-keys" Jul 29 04:40:57.826691 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.826596 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3524547f-2c7a-47ec-bebd-30bf3a7072b8" containerName="revoke-keys" Jul 29 04:40:57.826747 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.826695 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3524547f-2c7a-47ec-bebd-30bf3a7072b8" containerName="revoke-keys" Jul 29 04:40:57.830006 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.829952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.833215 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.833190 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a32259-serving-cert\"" Jul 29 04:40:57.841741 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.841720 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5"] Jul 29 04:40:57.871245 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.871216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:57.893379 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.893357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaee0f5-2d92-4120-8214-1a9859e52862-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.893502 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.893416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaee0f5-2d92-4120-8214-1a9859e52862-maas-api-tls\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.893552 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.893492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4hs\" (UniqueName: \"kubernetes.io/projected/aeaee0f5-2d92-4120-8214-1a9859e52862-kube-api-access-lv4hs\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.994439 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.994399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4hs\" (UniqueName: \"kubernetes.io/projected/aeaee0f5-2d92-4120-8214-1a9859e52862-kube-api-access-lv4hs\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.994597 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.994508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaee0f5-2d92-4120-8214-1a9859e52862-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.994597 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.994564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaee0f5-2d92-4120-8214-1a9859e52862-maas-api-tls\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.995215 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.995193 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaee0f5-2d92-4120-8214-1a9859e52862-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:57.997091 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:57.997075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaee0f5-2d92-4120-8214-1a9859e52862-maas-api-tls\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:58.004308 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:58.004289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4hs\" (UniqueName: \"kubernetes.io/projected/aeaee0f5-2d92-4120-8214-1a9859e52862-kube-api-access-lv4hs\") pod \"maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:58.028690 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:58.028661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:58.142637 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:58.142614 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:58.284252 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:58.284230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5"] Jul 29 04:40:58.285823 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:40:58.285791 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaee0f5_2d92_4120_8214_1a9859e52862.slice/crio-6199bd3d0f6bca6c164ba517a4c9fdd7f52c922a6767b8aa41365211832d390e WatchSource:0}: Error finding container 6199bd3d0f6bca6c164ba517a4c9fdd7f52c922a6767b8aa41365211832d390e: Status 404 returned error can't find the container with id 6199bd3d0f6bca6c164ba517a4c9fdd7f52c922a6767b8aa41365211832d390e Jul 29 04:40:58.287364 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:58.287346 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:40:58.343744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:58.343704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:58.429494 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:58.429418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:40:59.064652 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:59.064559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" event={"ID":"aeaee0f5-2d92-4120-8214-1a9859e52862","Type":"ContainerStarted","Data":"37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232"} Jul 29 04:40:59.064652 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:59.064613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" event={"ID":"aeaee0f5-2d92-4120-8214-1a9859e52862","Type":"ContainerStarted","Data":"6199bd3d0f6bca6c164ba517a4c9fdd7f52c922a6767b8aa41365211832d390e"} Jul 29 04:40:59.064652 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:59.064640 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:40:59.083080 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:40:59.083038 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" podStartSLOduration=1.7124349589999999 podStartE2EDuration="2.083022496s" totalImagesPullingTime="370.587537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:40:59.08120193 +0000 UTC m=+2741.373298293" watchObservedRunningTime="2026-07-29 04:40:59.083022496 +0000 UTC m=+2741.375118858" Jul 29 04:41:05.078841 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:41:05.078813 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:41:22.567247 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:41:22.567218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:41:26.472109 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:41:26.472075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:41:53.950799 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:41:53.950769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:32.159766 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:32.159732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:33.055312 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:33.055286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:43.655403 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:43.655375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:43.786548 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:43.786518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:43.877155 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:43.877091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:43.936203 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:43.936138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:44.296176 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:44.296098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:50.083898 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:50.083854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:50.138937 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:50.138911 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:50.348564 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:50.348491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:50.563436 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:50.563408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:42:50.617522 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:42:50.617450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:01.079531 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.079481 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5"] Jul 29 04:43:01.080038 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.079763 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" podUID="aeaee0f5-2d92-4120-8214-1a9859e52862" containerName="maas-api" containerID="cri-o://37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232" gracePeriod=30 Jul 29 04:43:01.104127 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.104104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:01.167930 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.167862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:01.225130 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.225100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:01.277943 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.277915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:01.340211 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.340159 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:43:01.442807 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.442773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaee0f5-2d92-4120-8214-1a9859e52862-maas-api-tls\") pod \"aeaee0f5-2d92-4120-8214-1a9859e52862\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " Jul 29 04:43:01.443019 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.442822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaee0f5-2d92-4120-8214-1a9859e52862-openshift-service-ca\") pod \"aeaee0f5-2d92-4120-8214-1a9859e52862\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " Jul 29 04:43:01.443019 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.442848 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv4hs\" (UniqueName: \"kubernetes.io/projected/aeaee0f5-2d92-4120-8214-1a9859e52862-kube-api-access-lv4hs\") pod \"aeaee0f5-2d92-4120-8214-1a9859e52862\" (UID: \"aeaee0f5-2d92-4120-8214-1a9859e52862\") " Jul 29 04:43:01.443255 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.443228 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeaee0f5-2d92-4120-8214-1a9859e52862-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aeaee0f5-2d92-4120-8214-1a9859e52862" (UID: "aeaee0f5-2d92-4120-8214-1a9859e52862"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:43:01.445162 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.445137 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeaee0f5-2d92-4120-8214-1a9859e52862-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aeaee0f5-2d92-4120-8214-1a9859e52862" (UID: "aeaee0f5-2d92-4120-8214-1a9859e52862"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:43:01.445273 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.445160 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeaee0f5-2d92-4120-8214-1a9859e52862-kube-api-access-lv4hs" (OuterVolumeSpecName: "kube-api-access-lv4hs") pod "aeaee0f5-2d92-4120-8214-1a9859e52862" (UID: "aeaee0f5-2d92-4120-8214-1a9859e52862"). InnerVolumeSpecName "kube-api-access-lv4hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:43:01.544193 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.544153 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaee0f5-2d92-4120-8214-1a9859e52862-maas-api-tls\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:43:01.544193 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.544183 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaee0f5-2d92-4120-8214-1a9859e52862-openshift-service-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:43:01.544193 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.544197 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4hs\" (UniqueName: \"kubernetes.io/projected/aeaee0f5-2d92-4120-8214-1a9859e52862-kube-api-access-lv4hs\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:43:01.688957 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.688931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:01.752750 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.752720 2576 generic.go:356] "Generic (PLEG): container finished" podID="aeaee0f5-2d92-4120-8214-1a9859e52862" containerID="37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232" exitCode=0 Jul 29 04:43:01.752911 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.752770 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" Jul 29 04:43:01.752911 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.752812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" event={"ID":"aeaee0f5-2d92-4120-8214-1a9859e52862","Type":"ContainerDied","Data":"37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232"} Jul 29 04:43:01.752911 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.752862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5" event={"ID":"aeaee0f5-2d92-4120-8214-1a9859e52862","Type":"ContainerDied","Data":"6199bd3d0f6bca6c164ba517a4c9fdd7f52c922a6767b8aa41365211832d390e"} Jul 29 04:43:01.752911 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.752908 2576 scope.go:117] "RemoveContainer" containerID="37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232" Jul 29 04:43:01.763460 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.763445 2576 scope.go:117] "RemoveContainer" containerID="37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232" Jul 29 04:43:01.763678 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:43:01.763660 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232\": container with ID starting with 37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232 not found: ID does not exist" containerID="37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232" Jul 29 04:43:01.763717 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.763687 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232"} err="failed to get container status \"37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232\": rpc error: code = NotFound desc = could not find container \"37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232\": container with ID starting with 37a61aa9d80a83da67c27fe11295f82473ee9e2a44a6ac08c6429844a8e51232 not found: ID does not exist" Jul 29 04:43:01.774133 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.774106 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5"] Jul 29 04:43:01.779544 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:01.779519 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a32259-c96ddd75c-2j8h5"] Jul 29 04:43:02.196979 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:02.196948 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeaee0f5-2d92-4120-8214-1a9859e52862" path="/var/lib/kubelet/pods/aeaee0f5-2d92-4120-8214-1a9859e52862/volumes" Jul 29 04:43:07.500505 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:07.500472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:07.553546 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:07.553520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:07.780301 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:07.780216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:07.996370 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:07.996336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:08.328369 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:08.328338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:08.646265 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:08.646236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:09.270961 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:09.270932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:11.178683 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:11.178650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:11.250673 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:11.250640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:11.320139 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:11.320113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:11.531840 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:11.531755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:11.744116 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:11.744092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:11.840560 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:11.840485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:15.733459 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:15.733423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:36.168743 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:36.168703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:36.625253 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:36.625220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:36.930265 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:36.930220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:38.058700 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:38.058672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:39.333930 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:39.333877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:39.416345 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:39.416319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:39.478239 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:39.478207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:39.935641 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:39.935614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:40.116170 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.116139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:40.200236 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.200164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:43:40.245201 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.245170 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm"] Jul 29 04:43:40.245643 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.245624 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeaee0f5-2d92-4120-8214-1a9859e52862" containerName="maas-api" Jul 29 04:43:40.245643 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.245644 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeaee0f5-2d92-4120-8214-1a9859e52862" containerName="maas-api" Jul 29 04:43:40.245796 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.245753 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeaee0f5-2d92-4120-8214-1a9859e52862" containerName="maas-api" Jul 29 04:43:40.248940 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.248919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.251416 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.251394 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a099ac\"" Jul 29 04:43:40.251523 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.251397 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a099ac-dockercfg-z868j\"" Jul 29 04:43:40.259590 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.259570 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm"] Jul 29 04:43:40.381955 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.381925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55694b0-ebba-45d3-acfd-7de93fcd1d24-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.381955 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.381958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmck\" (UniqueName: \"kubernetes.io/projected/d55694b0-ebba-45d3-acfd-7de93fcd1d24-kube-api-access-7xmck\") pod \"payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.482618 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.482535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55694b0-ebba-45d3-acfd-7de93fcd1d24-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.482618 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.482570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmck\" (UniqueName: \"kubernetes.io/projected/d55694b0-ebba-45d3-acfd-7de93fcd1d24-kube-api-access-7xmck\") pod \"payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.483238 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.483219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55694b0-ebba-45d3-acfd-7de93fcd1d24-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.494029 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.494000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmck\" (UniqueName: \"kubernetes.io/projected/d55694b0-ebba-45d3-acfd-7de93fcd1d24-kube-api-access-7xmck\") pod \"payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.560821 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.560793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:40.698917 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.698852 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm"] Jul 29 04:43:40.701948 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:43:40.701919 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55694b0_ebba_45d3_acfd_7de93fcd1d24.slice/crio-e1b72c44a2b43badcae11fb38a60bb723db8485d58a74f4b8698f4f20fa65872 WatchSource:0}: Error finding container e1b72c44a2b43badcae11fb38a60bb723db8485d58a74f4b8698f4f20fa65872: Status 404 returned error can't find the container with id e1b72c44a2b43badcae11fb38a60bb723db8485d58a74f4b8698f4f20fa65872 Jul 29 04:43:40.964647 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:40.964613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" event={"ID":"d55694b0-ebba-45d3-acfd-7de93fcd1d24","Type":"ContainerStarted","Data":"e1b72c44a2b43badcae11fb38a60bb723db8485d58a74f4b8698f4f20fa65872"} Jul 29 04:43:42.978773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:42.978727 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" event={"ID":"d55694b0-ebba-45d3-acfd-7de93fcd1d24","Type":"ContainerStarted","Data":"1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819"} Jul 29 04:43:42.979181 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:42.978822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:43:42.999954 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:42.999910 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" podStartSLOduration=1.518421223 podStartE2EDuration="2.9998712s" totalImagesPullingTime="1.481449977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:42.998940486 +0000 UTC m=+2905.291036849" watchObservedRunningTime="2026-07-29 04:43:42.9998712 +0000 UTC m=+2905.291967562" Jul 29 04:43:53.987096 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:43:53.987066 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:44:04.320085 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:04.320057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:22.657548 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:22.657513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:22.718752 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:22.718716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:22.814108 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:22.814080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:23.195331 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:23.195302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:28.998169 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:28.998098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:29.099774 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:29.099732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:29.234740 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:29.234712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:29.493592 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:29.493555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:45.350225 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.350147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:45.378499 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.378468 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm"] Jul 29 04:44:45.378742 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.378716 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" podUID="d55694b0-ebba-45d3-acfd-7de93fcd1d24" containerName="payload-processing" containerID="cri-o://1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819" gracePeriod=30 Jul 29 04:44:45.413599 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.413572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:45.473462 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.473426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:45.527800 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.527689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:45.642875 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.642850 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:44:45.740192 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.740160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xmck\" (UniqueName: \"kubernetes.io/projected/d55694b0-ebba-45d3-acfd-7de93fcd1d24-kube-api-access-7xmck\") pod \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " Jul 29 04:44:45.740355 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.740293 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55694b0-ebba-45d3-acfd-7de93fcd1d24-plugins-config-volume\") pod \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\" (UID: \"d55694b0-ebba-45d3-acfd-7de93fcd1d24\") " Jul 29 04:44:45.740643 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.740620 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55694b0-ebba-45d3-acfd-7de93fcd1d24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d55694b0-ebba-45d3-acfd-7de93fcd1d24" (UID: "d55694b0-ebba-45d3-acfd-7de93fcd1d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:44:45.742538 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.742507 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55694b0-ebba-45d3-acfd-7de93fcd1d24-kube-api-access-7xmck" (OuterVolumeSpecName: "kube-api-access-7xmck") pod "d55694b0-ebba-45d3-acfd-7de93fcd1d24" (UID: "d55694b0-ebba-45d3-acfd-7de93fcd1d24"). InnerVolumeSpecName "kube-api-access-7xmck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:44:45.840875 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.840850 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55694b0-ebba-45d3-acfd-7de93fcd1d24-plugins-config-volume\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:44:45.840875 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.840872 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmck\" (UniqueName: \"kubernetes.io/projected/d55694b0-ebba-45d3-acfd-7de93fcd1d24-kube-api-access-7xmck\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:44:45.891449 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:45.891384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:46.334773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.334744 2576 generic.go:356] "Generic (PLEG): container finished" podID="d55694b0-ebba-45d3-acfd-7de93fcd1d24" containerID="1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819" exitCode=0 Jul 29 04:44:46.334960 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.334797 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" Jul 29 04:44:46.334960 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.334831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" event={"ID":"d55694b0-ebba-45d3-acfd-7de93fcd1d24","Type":"ContainerDied","Data":"1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819"} Jul 29 04:44:46.334960 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.334908 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm" event={"ID":"d55694b0-ebba-45d3-acfd-7de93fcd1d24","Type":"ContainerDied","Data":"e1b72c44a2b43badcae11fb38a60bb723db8485d58a74f4b8698f4f20fa65872"} Jul 29 04:44:46.334960 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.334936 2576 scope.go:117] "RemoveContainer" containerID="1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819" Jul 29 04:44:46.344294 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.344278 2576 scope.go:117] "RemoveContainer" containerID="1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819" Jul 29 04:44:46.344549 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:44:46.344531 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819\": container with ID starting with 1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819 not found: ID does not exist" containerID="1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819" Jul 29 04:44:46.344601 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.344558 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819"} err="failed to get container status \"1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819\": rpc error: code = NotFound desc = could not find container \"1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819\": container with ID starting with 1c0fbeb718944b00745ec39ef0807894b689ffc19e507acf386475a391e8f819 not found: ID does not exist" Jul 29 04:44:46.351161 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.351140 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm"] Jul 29 04:44:46.354741 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:46.354721 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a099ac-6fd48c9c96-m6wpm"] Jul 29 04:44:48.197748 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:48.197711 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55694b0-ebba-45d3-acfd-7de93fcd1d24" path="/var/lib/kubelet/pods/d55694b0-ebba-45d3-acfd-7de93fcd1d24/volumes" Jul 29 04:44:51.539646 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:51.539613 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:51.640094 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:51.640061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:51.798575 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:51.798495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:52.027675 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:52.027641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:52.365655 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:52.365621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:52.651540 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:52.651506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:53.005246 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:53.005121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:55.723610 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:55.723576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:55.813033 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:55.813005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:55.874055 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:55.874028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:56.039762 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:56.039677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:56.490991 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:56.490961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:44:56.578701 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:44:56.578669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:18.411637 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:18.411613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:45:18.423066 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:18.423045 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:45:20.699032 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:20.698998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:21.022769 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:21.022693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:21.306290 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:21.306206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:21.447287 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:21.447253 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:23.888371 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:23.888305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:23.987062 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:23.987030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:24.119239 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.119212 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6"] Jul 29 04:45:24.119619 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.119607 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55694b0-ebba-45d3-acfd-7de93fcd1d24" containerName="payload-processing" Jul 29 04:45:24.119663 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.119622 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55694b0-ebba-45d3-acfd-7de93fcd1d24" containerName="payload-processing" Jul 29 04:45:24.119708 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.119700 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55694b0-ebba-45d3-acfd-7de93fcd1d24" containerName="payload-processing" Jul 29 04:45:24.123915 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.123876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.126120 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.126101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b550f6-serving-cert\"" Jul 29 04:45:24.130458 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.130437 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6"] Jul 29 04:45:24.154905 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.154825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:24.162105 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.162074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48447314-066a-4e17-a92d-53cabf5487b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.162191 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.162131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqwrn\" (UniqueName: \"kubernetes.io/projected/48447314-066a-4e17-a92d-53cabf5487b5-kube-api-access-zqwrn\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.162245 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.162208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.263190 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.263160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48447314-066a-4e17-a92d-53cabf5487b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.263353 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.263207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqwrn\" (UniqueName: \"kubernetes.io/projected/48447314-066a-4e17-a92d-53cabf5487b5-kube-api-access-zqwrn\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.263353 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.263304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.263487 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:45:24.263464 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-serving-cert: secret "maas-api-e2e-shared-b-b550f6-serving-cert" not found Jul 29 04:45:24.263568 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:45:24.263552 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls podName:48447314-066a-4e17-a92d-53cabf5487b5 nodeName:}" failed. No retries permitted until 2026-07-29 04:45:24.763528253 +0000 UTC m=+3007.055624595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls") pod "maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" (UID: "48447314-066a-4e17-a92d-53cabf5487b5") : secret "maas-api-e2e-shared-b-b550f6-serving-cert" not found Jul 29 04:45:24.263870 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.263847 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48447314-066a-4e17-a92d-53cabf5487b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.278184 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.278154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqwrn\" (UniqueName: \"kubernetes.io/projected/48447314-066a-4e17-a92d-53cabf5487b5-kube-api-access-zqwrn\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.396133 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.396102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:24.729840 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.729807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:24.768574 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.768543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.771064 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.771041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:24.831472 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.831448 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:45:24.855348 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.855317 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp"] Jul 29 04:45:24.859249 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.859231 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:24.861597 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.861571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-b550f6\"" Jul 29 04:45:24.861747 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.861729 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-b550f6-dockercfg-fpcrl\"" Jul 29 04:45:24.867134 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.867112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp"] Jul 29 04:45:24.970108 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.970075 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2snt\" (UniqueName: \"kubernetes.io/projected/d7885895-2bc7-459e-8b51-9bc0b290aea3-kube-api-access-k2snt\") pod \"payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:24.970470 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:24.970126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7885895-2bc7-459e-8b51-9bc0b290aea3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:25.038149 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.038079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:25.070866 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.070839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2snt\" (UniqueName: \"kubernetes.io/projected/d7885895-2bc7-459e-8b51-9bc0b290aea3-kube-api-access-k2snt\") pod \"payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:25.071017 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.070910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7885895-2bc7-459e-8b51-9bc0b290aea3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:25.071554 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.071536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7885895-2bc7-459e-8b51-9bc0b290aea3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:25.086323 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.086290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2snt\" (UniqueName: \"kubernetes.io/projected/d7885895-2bc7-459e-8b51-9bc0b290aea3-kube-api-access-k2snt\") pod \"payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:25.170732 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.170700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:25.173686 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.173660 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6"] Jul 29 04:45:25.175406 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:45:25.175376 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48447314_066a_4e17_a92d_53cabf5487b5.slice/crio-bd2b17dc9e59ade50d31977edf664bb946e6125e8af0ec92ae47d6c4dfe1e160 WatchSource:0}: Error finding container bd2b17dc9e59ade50d31977edf664bb946e6125e8af0ec92ae47d6c4dfe1e160: Status 404 returned error can't find the container with id bd2b17dc9e59ade50d31977edf664bb946e6125e8af0ec92ae47d6c4dfe1e160 Jul 29 04:45:25.515752 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.515714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp"] Jul 29 04:45:25.517170 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:45:25.517145 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7885895_2bc7_459e_8b51_9bc0b290aea3.slice/crio-3e172b57ad379617944335970df7edcbf34aa397c9d1f58b7bcfd527df3b7b07 WatchSource:0}: Error finding container 3e172b57ad379617944335970df7edcbf34aa397c9d1f58b7bcfd527df3b7b07: Status 404 returned error can't find the container with id 3e172b57ad379617944335970df7edcbf34aa397c9d1f58b7bcfd527df3b7b07 Jul 29 04:45:25.550840 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.550813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" event={"ID":"d7885895-2bc7-459e-8b51-9bc0b290aea3","Type":"ContainerStarted","Data":"3e172b57ad379617944335970df7edcbf34aa397c9d1f58b7bcfd527df3b7b07"} Jul 29 04:45:25.552091 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:25.552070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" event={"ID":"48447314-066a-4e17-a92d-53cabf5487b5","Type":"ContainerStarted","Data":"bd2b17dc9e59ade50d31977edf664bb946e6125e8af0ec92ae47d6c4dfe1e160"} Jul 29 04:45:26.559087 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:26.559030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" event={"ID":"48447314-066a-4e17-a92d-53cabf5487b5","Type":"ContainerStarted","Data":"cbe4ded54a0bebcde25701bdaccabfbfc568ce477fcda5dce015d631011e1162"} Jul 29 04:45:26.559528 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:26.559251 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:26.560651 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:26.560620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" event={"ID":"d7885895-2bc7-459e-8b51-9bc0b290aea3","Type":"ContainerStarted","Data":"2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc"} Jul 29 04:45:26.560783 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:26.560684 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:26.575655 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:26.575612 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" podStartSLOduration=1.833796509 podStartE2EDuration="2.575600836s" totalImagesPullingTime="741.804327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:45:26.574725473 +0000 UTC m=+3008.866821846" watchObservedRunningTime="2026-07-29 04:45:26.575600836 +0000 UTC m=+3008.867697197" Jul 29 04:45:26.591715 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:26.591672 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" podStartSLOduration=2.5916588149999997 podStartE2EDuration="2.591658815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:45:26.589194632 +0000 UTC m=+3008.881290994" watchObservedRunningTime="2026-07-29 04:45:26.591658815 +0000 UTC m=+3008.883755177" Jul 29 04:45:32.572760 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:32.572732 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 04:45:37.568329 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:37.568300 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 04:45:48.863622 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:45:48.863590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:46:03.351517 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:03.351488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:46:25.817015 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:25.816938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:46:26.953758 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:26.953724 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:46:30.315844 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.315805 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-th6l9"] Jul 29 04:46:30.319341 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.319322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:30.321859 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.321841 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-t5ljb\"" Jul 29 04:46:30.327254 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.327230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-th6l9"] Jul 29 04:46:30.443565 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.443532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjffk\" (UniqueName: \"kubernetes.io/projected/4d5ff045-1926-4486-b413-1eade63f20e1-kube-api-access-wjffk\") pod \"kuadrant-operator-catalog-th6l9\" (UID: \"4d5ff045-1926-4486-b413-1eade63f20e1\") " pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:30.544943 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.544904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjffk\" (UniqueName: \"kubernetes.io/projected/4d5ff045-1926-4486-b413-1eade63f20e1-kube-api-access-wjffk\") pod \"kuadrant-operator-catalog-th6l9\" (UID: \"4d5ff045-1926-4486-b413-1eade63f20e1\") " pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:30.553438 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.553412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjffk\" (UniqueName: \"kubernetes.io/projected/4d5ff045-1926-4486-b413-1eade63f20e1-kube-api-access-wjffk\") pod \"kuadrant-operator-catalog-th6l9\" (UID: \"4d5ff045-1926-4486-b413-1eade63f20e1\") " pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:30.630413 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.630377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:30.765307 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.765283 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-th6l9"] Jul 29 04:46:30.767182 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:46:30.767156 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5ff045_1926_4486_b413_1eade63f20e1.slice/crio-4be74f2b70b9d411a75a31b406d79e4c2bffbcd601e56442eb6782e312ad071e WatchSource:0}: Error finding container 4be74f2b70b9d411a75a31b406d79e4c2bffbcd601e56442eb6782e312ad071e: Status 404 returned error can't find the container with id 4be74f2b70b9d411a75a31b406d79e4c2bffbcd601e56442eb6782e312ad071e Jul 29 04:46:30.768940 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.768923 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:46:30.912184 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:30.912098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" event={"ID":"4d5ff045-1926-4486-b413-1eade63f20e1","Type":"ContainerStarted","Data":"4be74f2b70b9d411a75a31b406d79e4c2bffbcd601e56442eb6782e312ad071e"} Jul 29 04:46:32.925736 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:32.925704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" event={"ID":"4d5ff045-1926-4486-b413-1eade63f20e1","Type":"ContainerStarted","Data":"72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a"} Jul 29 04:46:32.941303 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:32.941256 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" podStartSLOduration=1.5525676050000001 podStartE2EDuration="2.941242873s" totalImagesPullingTime="1.388675268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:46:32.938811513 +0000 UTC m=+3075.230907886" watchObservedRunningTime="2026-07-29 04:46:32.941242873 +0000 UTC m=+3075.233339235" Jul 29 04:46:34.463168 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:34.463133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:46:40.630803 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:40.630766 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:40.631235 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:40.630864 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:40.653059 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:40.653034 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:40.997139 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:40.997055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:41.031419 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:41.031386 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-th6l9"] Jul 29 04:46:42.986611 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:42.986564 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" podUID="4d5ff045-1926-4486-b413-1eade63f20e1" containerName="registry-server" containerID="cri-o://72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a" gracePeriod=2 Jul 29 04:46:43.250844 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.250788 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:43.361116 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.361078 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjffk\" (UniqueName: \"kubernetes.io/projected/4d5ff045-1926-4486-b413-1eade63f20e1-kube-api-access-wjffk\") pod \"4d5ff045-1926-4486-b413-1eade63f20e1\" (UID: \"4d5ff045-1926-4486-b413-1eade63f20e1\") " Jul 29 04:46:43.363237 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.363209 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5ff045-1926-4486-b413-1eade63f20e1-kube-api-access-wjffk" (OuterVolumeSpecName: "kube-api-access-wjffk") pod "4d5ff045-1926-4486-b413-1eade63f20e1" (UID: "4d5ff045-1926-4486-b413-1eade63f20e1"). InnerVolumeSpecName "kube-api-access-wjffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:46:43.462488 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.462459 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjffk\" (UniqueName: \"kubernetes.io/projected/4d5ff045-1926-4486-b413-1eade63f20e1-kube-api-access-wjffk\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:46:43.993324 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.993292 2576 generic.go:356] "Generic (PLEG): container finished" podID="4d5ff045-1926-4486-b413-1eade63f20e1" containerID="72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a" exitCode=0 Jul 29 04:46:43.993798 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.993335 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" Jul 29 04:46:43.993798 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.993386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" event={"ID":"4d5ff045-1926-4486-b413-1eade63f20e1","Type":"ContainerDied","Data":"72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a"} Jul 29 04:46:43.993798 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.993445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-th6l9" event={"ID":"4d5ff045-1926-4486-b413-1eade63f20e1","Type":"ContainerDied","Data":"4be74f2b70b9d411a75a31b406d79e4c2bffbcd601e56442eb6782e312ad071e"} Jul 29 04:46:43.993798 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:43.993483 2576 scope.go:117] "RemoveContainer" containerID="72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a" Jul 29 04:46:44.004071 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:44.004051 2576 scope.go:117] "RemoveContainer" containerID="72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a" Jul 29 04:46:44.004317 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:46:44.004298 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a\": container with ID starting with 72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a not found: ID does not exist" containerID="72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a" Jul 29 04:46:44.004383 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:44.004329 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a"} err="failed to get container status \"72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a\": rpc error: code = NotFound desc = could not find container \"72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a\": container with ID starting with 72f90ff85c8887b7e8a7fa60d539ff45badf4a0a0c3460a8a183ea987f86128a not found: ID does not exist" Jul 29 04:46:44.014143 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:44.014120 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-th6l9"] Jul 29 04:46:44.017851 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:44.017832 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-th6l9"] Jul 29 04:46:44.198260 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:46:44.198233 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5ff045-1926-4486-b413-1eade63f20e1" path="/var/lib/kubelet/pods/4d5ff045-1926-4486-b413-1eade63f20e1/volumes" Jul 29 04:47:57.801008 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:47:57.800968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:48:14.500594 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:48:14.500536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:49:19.244875 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:49:19.244788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:49:56.616383 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:49:56.616339 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:50:18.449235 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:50:18.449207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:50:18.461059 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:50:18.461040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:50:21.635899 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:50:21.635851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:51:29.430204 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:51:29.430174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:51:30.963025 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:51:30.962984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:52:03.458331 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:52:03.458297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:52:43.957138 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:52:43.957104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:54:26.162828 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:54:26.162796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:54:27.032830 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:54:27.032798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:54:34.350871 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:54:34.350819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:54:54.665569 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:54:54.665524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:55:18.485460 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:55:18.485427 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:55:18.499383 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:55:18.499361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 04:56:36.329115 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:36.329070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:56:37.649585 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:37.649557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:56:44.586709 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.586631 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7677-140502348539056"] Jul 29 04:56:44.587355 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.587117 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5ff045-1926-4486-b413-1eade63f20e1" containerName="registry-server" Jul 29 04:56:44.587355 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.587133 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5ff045-1926-4486-b413-1eade63f20e1" containerName="registry-server" Jul 29 04:56:44.587355 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.587192 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5ff045-1926-4486-b413-1eade63f20e1" containerName="registry-server" Jul 29 04:56:44.590314 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.590297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348539056" Jul 29 04:56:44.593267 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.593249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-k8w2g\"" Jul 29 04:56:44.598619 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.598594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7677-140502348539056"] Jul 29 04:56:44.709217 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.709184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgfj2\" (UniqueName: \"kubernetes.io/projected/cdfd39fb-9b6f-4243-8b27-1ee84915cd13-kube-api-access-lgfj2\") pod \"test-curl-7677-140502348539056\" (UID: \"cdfd39fb-9b6f-4243-8b27-1ee84915cd13\") " pod="opendatahub/test-curl-7677-140502348539056" Jul 29 04:56:44.810213 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.810186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgfj2\" (UniqueName: \"kubernetes.io/projected/cdfd39fb-9b6f-4243-8b27-1ee84915cd13-kube-api-access-lgfj2\") pod \"test-curl-7677-140502348539056\" (UID: \"cdfd39fb-9b6f-4243-8b27-1ee84915cd13\") " pod="opendatahub/test-curl-7677-140502348539056" Jul 29 04:56:44.822968 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.822946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgfj2\" (UniqueName: \"kubernetes.io/projected/cdfd39fb-9b6f-4243-8b27-1ee84915cd13-kube-api-access-lgfj2\") pod \"test-curl-7677-140502348539056\" (UID: \"cdfd39fb-9b6f-4243-8b27-1ee84915cd13\") " pod="opendatahub/test-curl-7677-140502348539056" Jul 29 04:56:44.900854 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:44.900834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348539056" Jul 29 04:56:45.041903 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:45.041830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7677-140502348539056"] Jul 29 04:56:45.044772 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:56:45.044744 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdfd39fb_9b6f_4243_8b27_1ee84915cd13.slice/crio-af41295e131002487bb4bdfcb2da0128ca914535f75690ba500644e678878f37 WatchSource:0}: Error finding container af41295e131002487bb4bdfcb2da0128ca914535f75690ba500644e678878f37: Status 404 returned error can't find the container with id af41295e131002487bb4bdfcb2da0128ca914535f75690ba500644e678878f37 Jul 29 04:56:45.046413 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:45.046395 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:56:45.329209 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:45.329141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7677-140502348539056" event={"ID":"cdfd39fb-9b6f-4243-8b27-1ee84915cd13","Type":"ContainerStarted","Data":"af41295e131002487bb4bdfcb2da0128ca914535f75690ba500644e678878f37"} Jul 29 04:56:46.336342 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:46.336313 2576 generic.go:356] "Generic (PLEG): container finished" podID="cdfd39fb-9b6f-4243-8b27-1ee84915cd13" containerID="eedfc55a3cb107bd4ba8c5d84323183344a62fac63794c5d1608e93ce816e004" exitCode=0 Jul 29 04:56:46.336746 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:46.336402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7677-140502348539056" event={"ID":"cdfd39fb-9b6f-4243-8b27-1ee84915cd13","Type":"ContainerDied","Data":"eedfc55a3cb107bd4ba8c5d84323183344a62fac63794c5d1608e93ce816e004"} Jul 29 04:56:47.487636 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:47.487613 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348539056" Jul 29 04:56:47.507273 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:47.507249 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7677-140502348539056_cdfd39fb-9b6f-4243-8b27-1ee84915cd13/test-curl-7677-140502348539056/0.log" Jul 29 04:56:47.535499 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:47.535463 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7677-140502348539056"] Jul 29 04:56:47.537822 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:47.537803 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7677-140502348539056"] Jul 29 04:56:47.636008 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:47.635985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgfj2\" (UniqueName: \"kubernetes.io/projected/cdfd39fb-9b6f-4243-8b27-1ee84915cd13-kube-api-access-lgfj2\") pod \"cdfd39fb-9b6f-4243-8b27-1ee84915cd13\" (UID: \"cdfd39fb-9b6f-4243-8b27-1ee84915cd13\") " Jul 29 04:56:47.638218 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:47.638198 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfd39fb-9b6f-4243-8b27-1ee84915cd13-kube-api-access-lgfj2" (OuterVolumeSpecName: "kube-api-access-lgfj2") pod "cdfd39fb-9b6f-4243-8b27-1ee84915cd13" (UID: "cdfd39fb-9b6f-4243-8b27-1ee84915cd13"). InnerVolumeSpecName "kube-api-access-lgfj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:56:47.737287 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:47.737260 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgfj2\" (UniqueName: \"kubernetes.io/projected/cdfd39fb-9b6f-4243-8b27-1ee84915cd13-kube-api-access-lgfj2\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:56:48.198683 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:48.198649 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdfd39fb-9b6f-4243-8b27-1ee84915cd13" path="/var/lib/kubelet/pods/cdfd39fb-9b6f-4243-8b27-1ee84915cd13/volumes" Jul 29 04:56:48.348263 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:48.348240 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348539056" Jul 29 04:56:48.348432 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:48.348289 2576 scope.go:117] "RemoveContainer" containerID="eedfc55a3cb107bd4ba8c5d84323183344a62fac63794c5d1608e93ce816e004" Jul 29 04:56:56.832608 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:56.832580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:56:57.151507 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:57.151468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:56:57.451319 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:56:57.451241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:00.178953 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:00.178919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:00.289812 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:00.289782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:00.395743 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:00.395714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:00.500060 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:00.499969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:00.867460 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:00.867366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:00.969591 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:00.969557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:06.070222 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:06.070183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:25.149777 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:25.149741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:25.824038 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:25.823976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:26.089095 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.089015 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb"] Jul 29 04:57:26.089518 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.089500 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfd39fb-9b6f-4243-8b27-1ee84915cd13" containerName="test-curl-7677-140502348539056" Jul 29 04:57:26.089608 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.089521 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfd39fb-9b6f-4243-8b27-1ee84915cd13" containerName="test-curl-7677-140502348539056" Jul 29 04:57:26.089668 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.089637 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfd39fb-9b6f-4243-8b27-1ee84915cd13" containerName="test-curl-7677-140502348539056" Jul 29 04:57:26.094103 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.094082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.096625 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.096604 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2cb3bf-openshift-default-dockercfg-bqzp2\"" Jul 29 04:57:26.103180 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.103158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb"] Jul 29 04:57:26.114857 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.114816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:26.146173 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-certs\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-envoy\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-data\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-podinfo\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istiod-ca-cert\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-credential-socket\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146537 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-socket\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146537 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sb6t\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-kube-api-access-8sb6t\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.146537 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.146433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-token\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247258 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-certs\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-envoy\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-data\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-podinfo\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istiod-ca-cert\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-credential-socket\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-socket\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sb6t\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-kube-api-access-8sb6t\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.247744 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-token\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.248335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.247800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-certs\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.248335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.248000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-credential-socket\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.248335 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.248303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istiod-ca-cert\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.248569 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.248390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-data\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.248569 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.248510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-socket\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.249919 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.249869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-envoy\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.250177 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.250156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-podinfo\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.256094 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.256075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-token\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.256259 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.256243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sb6t\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-kube-api-access-8sb6t\") pod \"e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.319068 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.319033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:26.406004 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.405975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:26.572083 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:26.572059 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb"] Jul 29 04:57:26.572729 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:57:26.572689 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6462b5a9_80a3_41eb_bc52_0cbfe7c64cdd.slice/crio-875d8b2b11be7b19ac895adefceb1cc53ada0ae21e5a4f7897c395dcc4dcd113 WatchSource:0}: Error finding container 875d8b2b11be7b19ac895adefceb1cc53ada0ae21e5a4f7897c395dcc4dcd113: Status 404 returned error can't find the container with id 875d8b2b11be7b19ac895adefceb1cc53ada0ae21e5a4f7897c395dcc4dcd113 Jul 29 04:57:27.572114 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:27.572075 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" event={"ID":"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd","Type":"ContainerStarted","Data":"875d8b2b11be7b19ac895adefceb1cc53ada0ae21e5a4f7897c395dcc4dcd113"} Jul 29 04:57:29.620139 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:29.620105 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:57:29.620469 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:29.620174 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:57:29.620469 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:29.620206 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:57:30.590565 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:30.590521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" event={"ID":"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd","Type":"ContainerStarted","Data":"6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795"} Jul 29 04:57:30.614347 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:30.614288 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" podStartSLOduration=1.569562328 podStartE2EDuration="4.614273835s" totalImagesPullingTime="3.044711507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:57:30.613859094 +0000 UTC m=+3732.905955453" watchObservedRunningTime="2026-07-29 04:57:30.614273835 +0000 UTC m=+3732.906370198" Jul 29 04:57:31.406127 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:31.406091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:31.411315 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:31.411288 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:31.595862 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:31.595831 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:31.596870 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:31.596847 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:32.550216 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:32.550179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:32.648321 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:32.648294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:32.785815 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:32.785781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:33.035189 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.035156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:33.364347 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.364268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:33.460252 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.460219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:57:33.480143 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.480099 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws"] Jul 29 04:57:33.484028 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.484005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.487464 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.487436 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2cb3bf\"" Jul 29 04:57:33.487759 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.487741 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2cb3bf-dockercfg-chv2f\"" Jul 29 04:57:33.498853 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.498829 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws"] Jul 29 04:57:33.615583 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.615505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6xk6\" (UniqueName: \"kubernetes.io/projected/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-kube-api-access-j6xk6\") pod \"payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.615583 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.615560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.716661 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.716624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6xk6\" (UniqueName: \"kubernetes.io/projected/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-kube-api-access-j6xk6\") pod \"payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.716868 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.716729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.717430 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.717405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.724582 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.724555 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6xk6\" (UniqueName: \"kubernetes.io/projected/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-kube-api-access-j6xk6\") pod \"payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.794488 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.794449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:33.936124 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:33.936101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws"] Jul 29 04:57:33.937797 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:57:33.937767 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e27bea_5e1f_4045_9a4c_2b4e44d3428c.slice/crio-93518c659f13d3fb2e6cb36338107b5f49148e114fce99b17c925ba84d517501 WatchSource:0}: Error finding container 93518c659f13d3fb2e6cb36338107b5f49148e114fce99b17c925ba84d517501: Status 404 returned error can't find the container with id 93518c659f13d3fb2e6cb36338107b5f49148e114fce99b17c925ba84d517501 Jul 29 04:57:34.613591 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:34.613557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" event={"ID":"66e27bea-5e1f-4045-9a4c-2b4e44d3428c","Type":"ContainerStarted","Data":"d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4"} Jul 29 04:57:34.613773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:34.613609 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" event={"ID":"66e27bea-5e1f-4045-9a4c-2b4e44d3428c","Type":"ContainerStarted","Data":"93518c659f13d3fb2e6cb36338107b5f49148e114fce99b17c925ba84d517501"} Jul 29 04:57:34.613773 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:34.613631 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:34.628098 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:34.628038 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" podStartSLOduration=1.6280195119999998 podStartE2EDuration="1.628019512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:57:34.627536778 +0000 UTC m=+3736.919633140" watchObservedRunningTime="2026-07-29 04:57:34.628019512 +0000 UTC m=+3736.920115875" Jul 29 04:57:45.621404 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:45.621373 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 04:57:46.647371 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.647317 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:34398->10.132.0.63:15021: read: connection reset by peer" start-of-body= Jul 29 04:57:46.647755 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.647406 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:34398->10.132.0.63:15021: read: connection reset by peer" Jul 29 04:57:46.684386 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.684355 2576 generic.go:356] "Generic (PLEG): container finished" podID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerID="6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795" exitCode=0 Jul 29 04:57:46.684530 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.684403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" event={"ID":"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd","Type":"ContainerDied","Data":"6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795"} Jul 29 04:57:46.684854 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.684837 2576 scope.go:117] "RemoveContainer" containerID="6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795" Jul 29 04:57:46.687034 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.687002 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:57:46.687136 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.687076 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:57:46.687136 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:46.687106 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 04:57:47.698578 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:47.698537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" event={"ID":"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd","Type":"ContainerStarted","Data":"e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1"} Jul 29 04:57:48.406308 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:48.406271 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:48.406501 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:48.406482 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 04:57:48.406549 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:48.406528 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 04:57:49.407214 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:49.407178 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 04:57:49.407614 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:49.407254 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 04:57:50.406622 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:50.406582 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 04:57:50.406796 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:50.406655 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 04:57:51.406545 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:51.406510 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 04:57:51.406969 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:51.406569 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 04:57:52.410229 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:52.410202 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:52.410590 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:52.410530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:52.411247 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:52.411228 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 04:57:57.530897 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:57:57.530842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:04.500805 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:04.500776 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws_66e27bea-5e1f-4045-9a4c-2b4e44d3428c/payload-processing/0.log" Jul 29 04:58:09.672170 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:09.672080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:15.731815 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:15.731774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:49.448123 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:49.448090 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws_66e27bea-5e1f-4045-9a4c-2b4e44d3428c/payload-processing/0.log" Jul 29 04:58:49.728321 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:49.728223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:50.009033 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:50.008951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:50.366368 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:50.366279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:52.723234 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.723193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:52.806771 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.806743 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2"] Jul 29 04:58:52.810533 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.810513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.811247 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.811227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:52.813548 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.813528 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-79e09a-serving-cert\"" Jul 29 04:58:52.824127 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.824101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2"] Jul 29 04:58:52.826821 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.826789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mn9\" (UniqueName: \"kubernetes.io/projected/718443cc-e97c-4841-83ac-187a847f232c-kube-api-access-w7mn9\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.826975 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.826845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718443cc-e97c-4841-83ac-187a847f232c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.827038 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.826998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718443cc-e97c-4841-83ac-187a847f232c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.891147 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.891125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:52.928082 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.928052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mn9\" (UniqueName: \"kubernetes.io/projected/718443cc-e97c-4841-83ac-187a847f232c-kube-api-access-w7mn9\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.928234 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.928104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718443cc-e97c-4841-83ac-187a847f232c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.928234 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.928164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718443cc-e97c-4841-83ac-187a847f232c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.928771 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.928749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718443cc-e97c-4841-83ac-187a847f232c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.931839 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.931804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718443cc-e97c-4841-83ac-187a847f232c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.937895 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.937853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mn9\" (UniqueName: \"kubernetes.io/projected/718443cc-e97c-4841-83ac-187a847f232c-kube-api-access-w7mn9\") pod \"maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:52.992202 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:52.992129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:53.122107 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.122081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:53.264021 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.263993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2"] Jul 29 04:58:53.265303 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:58:53.265268 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718443cc_e97c_4841_83ac_187a847f232c.slice/crio-b37cff1db2b85101d80872e8dd525be78c5d3b0e0aefb4d80ad01e6e2096acc7 WatchSource:0}: Error finding container b37cff1db2b85101d80872e8dd525be78c5d3b0e0aefb4d80ad01e6e2096acc7: Status 404 returned error can't find the container with id b37cff1db2b85101d80872e8dd525be78c5d3b0e0aefb4d80ad01e6e2096acc7 Jul 29 04:58:53.409237 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.409189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:53.511823 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.511716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:58:53.537415 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.537384 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf"] Jul 29 04:58:53.542263 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.542237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.545847 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.545824 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-79e09a\"" Jul 29 04:58:53.546072 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.546057 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-79e09a-dockercfg-5cks5\"" Jul 29 04:58:53.554508 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.554484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf"] Jul 29 04:58:53.635250 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.635221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bkr\" (UniqueName: \"kubernetes.io/projected/49299e45-4572-40fb-a505-3f301ffe5491-kube-api-access-s8bkr\") pod \"payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.635446 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.635275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49299e45-4572-40fb-a505-3f301ffe5491-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.736422 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.736398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bkr\" (UniqueName: \"kubernetes.io/projected/49299e45-4572-40fb-a505-3f301ffe5491-kube-api-access-s8bkr\") pod \"payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.736708 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.736442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49299e45-4572-40fb-a505-3f301ffe5491-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.737241 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.737147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49299e45-4572-40fb-a505-3f301ffe5491-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.744713 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.744690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bkr\" (UniqueName: \"kubernetes.io/projected/49299e45-4572-40fb-a505-3f301ffe5491-kube-api-access-s8bkr\") pod \"payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.853836 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.853754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:53.997701 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:53.997667 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf"] Jul 29 04:58:53.997915 ip-10-0-140-47 kubenswrapper[2576]: W0729 04:58:53.997872 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49299e45_4572_40fb_a505_3f301ffe5491.slice/crio-694853c28b68050ee1b10d58208c1d9ced21601e9056e0ae5af39ebe7447e844 WatchSource:0}: Error finding container 694853c28b68050ee1b10d58208c1d9ced21601e9056e0ae5af39ebe7447e844: Status 404 returned error can't find the container with id 694853c28b68050ee1b10d58208c1d9ced21601e9056e0ae5af39ebe7447e844 Jul 29 04:58:54.084576 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.084546 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" event={"ID":"718443cc-e97c-4841-83ac-187a847f232c","Type":"ContainerStarted","Data":"782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830"} Jul 29 04:58:54.084702 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.084605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" event={"ID":"718443cc-e97c-4841-83ac-187a847f232c","Type":"ContainerStarted","Data":"b37cff1db2b85101d80872e8dd525be78c5d3b0e0aefb4d80ad01e6e2096acc7"} Jul 29 04:58:54.084702 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.084635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:58:54.086348 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.086327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" event={"ID":"49299e45-4572-40fb-a505-3f301ffe5491","Type":"ContainerStarted","Data":"7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285"} Jul 29 04:58:54.086462 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.086366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" event={"ID":"49299e45-4572-40fb-a505-3f301ffe5491","Type":"ContainerStarted","Data":"694853c28b68050ee1b10d58208c1d9ced21601e9056e0ae5af39ebe7447e844"} Jul 29 04:58:54.086462 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.086421 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:58:54.101862 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.101823 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" podStartSLOduration=1.6569734299999999 podStartE2EDuration="2.101809695s" totalImagesPullingTime="444.836265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:58:54.10033264 +0000 UTC m=+3816.392429029" watchObservedRunningTime="2026-07-29 04:58:54.101809695 +0000 UTC m=+3816.393906079" Jul 29 04:58:54.120332 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:58:54.120261 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" podStartSLOduration=1.120247583 podStartE2EDuration="1.120247583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:58:54.117857603 +0000 UTC m=+3816.409953966" watchObservedRunningTime="2026-07-29 04:58:54.120247583 +0000 UTC m=+3816.412343948" Jul 29 04:59:00.098499 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:00.098471 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:59:05.094492 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:05.094454 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:59:17.711449 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:17.711413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:23.376778 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.376746 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2"] Jul 29 04:59:23.377199 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.376961 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" podUID="718443cc-e97c-4841-83ac-187a847f232c" containerName="maas-api" containerID="cri-o://782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830" gracePeriod=30 Jul 29 04:59:23.403115 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.403088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:23.429931 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.429902 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf"] Jul 29 04:59:23.430159 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.430135 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" podUID="49299e45-4572-40fb-a505-3f301ffe5491" containerName="payload-processing" containerID="cri-o://7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285" gracePeriod=30 Jul 29 04:59:23.502638 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.502610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:23.565374 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.565348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:23.620914 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.620862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:23.695063 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.695039 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:59:23.726397 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.726369 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:59:23.786650 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.786623 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718443cc-e97c-4841-83ac-187a847f232c-maas-api-tls\") pod \"718443cc-e97c-4841-83ac-187a847f232c\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " Jul 29 04:59:23.786811 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.786666 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7mn9\" (UniqueName: \"kubernetes.io/projected/718443cc-e97c-4841-83ac-187a847f232c-kube-api-access-w7mn9\") pod \"718443cc-e97c-4841-83ac-187a847f232c\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " Jul 29 04:59:23.786811 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.786690 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8bkr\" (UniqueName: \"kubernetes.io/projected/49299e45-4572-40fb-a505-3f301ffe5491-kube-api-access-s8bkr\") pod \"49299e45-4572-40fb-a505-3f301ffe5491\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " Jul 29 04:59:23.786811 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.786764 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49299e45-4572-40fb-a505-3f301ffe5491-plugins-config-volume\") pod \"49299e45-4572-40fb-a505-3f301ffe5491\" (UID: \"49299e45-4572-40fb-a505-3f301ffe5491\") " Jul 29 04:59:23.787003 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.786842 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718443cc-e97c-4841-83ac-187a847f232c-openshift-service-ca\") pod \"718443cc-e97c-4841-83ac-187a847f232c\" (UID: \"718443cc-e97c-4841-83ac-187a847f232c\") " Jul 29 04:59:23.787238 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.787213 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49299e45-4572-40fb-a505-3f301ffe5491-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49299e45-4572-40fb-a505-3f301ffe5491" (UID: "49299e45-4572-40fb-a505-3f301ffe5491"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:59:23.787327 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.787235 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718443cc-e97c-4841-83ac-187a847f232c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "718443cc-e97c-4841-83ac-187a847f232c" (UID: "718443cc-e97c-4841-83ac-187a847f232c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:59:23.788952 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.788929 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49299e45-4572-40fb-a505-3f301ffe5491-kube-api-access-s8bkr" (OuterVolumeSpecName: "kube-api-access-s8bkr") pod "49299e45-4572-40fb-a505-3f301ffe5491" (UID: "49299e45-4572-40fb-a505-3f301ffe5491"). InnerVolumeSpecName "kube-api-access-s8bkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:59:23.789296 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.789276 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718443cc-e97c-4841-83ac-187a847f232c-kube-api-access-w7mn9" (OuterVolumeSpecName: "kube-api-access-w7mn9") pod "718443cc-e97c-4841-83ac-187a847f232c" (UID: "718443cc-e97c-4841-83ac-187a847f232c"). InnerVolumeSpecName "kube-api-access-w7mn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:59:23.789359 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.789277 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718443cc-e97c-4841-83ac-187a847f232c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "718443cc-e97c-4841-83ac-187a847f232c" (UID: "718443cc-e97c-4841-83ac-187a847f232c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:59:23.888630 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.888597 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49299e45-4572-40fb-a505-3f301ffe5491-plugins-config-volume\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:59:23.888630 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.888624 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718443cc-e97c-4841-83ac-187a847f232c-openshift-service-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:59:23.888630 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.888638 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718443cc-e97c-4841-83ac-187a847f232c-maas-api-tls\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:59:23.888849 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.888650 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mn9\" (UniqueName: \"kubernetes.io/projected/718443cc-e97c-4841-83ac-187a847f232c-kube-api-access-w7mn9\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:59:23.888849 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.888661 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8bkr\" (UniqueName: \"kubernetes.io/projected/49299e45-4572-40fb-a505-3f301ffe5491-kube-api-access-s8bkr\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 04:59:23.920052 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:23.919962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:24.266151 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.266120 2576 generic.go:356] "Generic (PLEG): container finished" podID="718443cc-e97c-4841-83ac-187a847f232c" containerID="782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830" exitCode=0 Jul 29 04:59:24.266338 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.266170 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" Jul 29 04:59:24.266338 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.266203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" event={"ID":"718443cc-e97c-4841-83ac-187a847f232c","Type":"ContainerDied","Data":"782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830"} Jul 29 04:59:24.266338 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.266253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2" event={"ID":"718443cc-e97c-4841-83ac-187a847f232c","Type":"ContainerDied","Data":"b37cff1db2b85101d80872e8dd525be78c5d3b0e0aefb4d80ad01e6e2096acc7"} Jul 29 04:59:24.266338 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.266275 2576 scope.go:117] "RemoveContainer" containerID="782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830" Jul 29 04:59:24.267724 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.267701 2576 generic.go:356] "Generic (PLEG): container finished" podID="49299e45-4572-40fb-a505-3f301ffe5491" containerID="7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285" exitCode=0 Jul 29 04:59:24.268110 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.267743 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" Jul 29 04:59:24.268110 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.267740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" event={"ID":"49299e45-4572-40fb-a505-3f301ffe5491","Type":"ContainerDied","Data":"7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285"} Jul 29 04:59:24.268110 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.267796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf" event={"ID":"49299e45-4572-40fb-a505-3f301ffe5491","Type":"ContainerDied","Data":"694853c28b68050ee1b10d58208c1d9ced21601e9056e0ae5af39ebe7447e844"} Jul 29 04:59:24.277226 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.277210 2576 scope.go:117] "RemoveContainer" containerID="782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830" Jul 29 04:59:24.277497 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:59:24.277481 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830\": container with ID starting with 782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830 not found: ID does not exist" containerID="782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830" Jul 29 04:59:24.277546 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.277506 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830"} err="failed to get container status \"782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830\": rpc error: code = NotFound desc = could not find container \"782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830\": container with ID starting with 782a97ded045d5ca5e88c8fc14f8bad9a15af2cb621ada7837803d79d1f5c830 not found: ID does not exist" Jul 29 04:59:24.277546 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.277521 2576 scope.go:117] "RemoveContainer" containerID="7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285" Jul 29 04:59:24.286802 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.286774 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2"] Jul 29 04:59:24.291010 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.290701 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-79e09a-5b4865bd5d-jjbb2"] Jul 29 04:59:24.291131 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.291098 2576 scope.go:117] "RemoveContainer" containerID="7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285" Jul 29 04:59:24.291650 ip-10-0-140-47 kubenswrapper[2576]: E0729 04:59:24.291629 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285\": container with ID starting with 7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285 not found: ID does not exist" containerID="7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285" Jul 29 04:59:24.291726 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.291661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285"} err="failed to get container status \"7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285\": rpc error: code = NotFound desc = could not find container \"7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285\": container with ID starting with 7596470e1fce53ebeeaa9395f172cd775b77e7dae6d1db63e771c31b6f20a285 not found: ID does not exist" Jul 29 04:59:24.297707 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.297688 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf"] Jul 29 04:59:24.301539 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:24.301519 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-79e09a-6958899658-dnblf"] Jul 29 04:59:26.197918 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:26.197870 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49299e45-4572-40fb-a505-3f301ffe5491" path="/var/lib/kubelet/pods/49299e45-4572-40fb-a505-3f301ffe5491/volumes" Jul 29 04:59:26.198327 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:26.198274 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="718443cc-e97c-4841-83ac-187a847f232c" path="/var/lib/kubelet/pods/718443cc-e97c-4841-83ac-187a847f232c/volumes" Jul 29 04:59:29.630232 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:29.630197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:29.690005 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:29.689981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:29.883391 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:29.883295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:30.116344 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:30.116310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:49.223598 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:49.223567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:49.290532 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:49.290480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:49.348116 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:49.348083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:49.408976 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:49.408951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:49.758434 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:49.758405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 04:59:54.788065 ip-10-0-140-47 kubenswrapper[2576]: I0729 04:59:54.788039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:04.437550 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:04.437506 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp"] Jul 29 05:00:04.441677 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:04.441650 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-ctvcp"] Jul 29 05:00:06.197699 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:06.197654 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="239d5efe-ea70-4477-9c15-dee88469b3af" path="/var/lib/kubelet/pods/239d5efe-ea70-4477-9c15-dee88469b3af/volumes" Jul 29 05:00:18.522715 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:18.522690 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 05:00:18.538297 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:18.538280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 05:00:18.691067 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:18.691041 2576 scope.go:117] "RemoveContainer" containerID="0b07c0a50a257d7cff931becec4bc2f1c8e6da15e1718c7ca70f37bb54555f00" Jul 29 05:00:31.037134 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:31.037105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:31.644964 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:31.644934 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:37.094584 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:37.094554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:39.526200 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:39.526162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:39.578506 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:39.578479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:50.999784 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:50.999750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:51.059363 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:51.059337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:51.357192 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:51.357097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:00:51.488938 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:00:51.488905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:02.133078 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.133051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:02.156311 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.156279 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws"] Jul 29 05:01:02.156785 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.156542 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" podUID="66e27bea-5e1f-4045-9a4c-2b4e44d3428c" containerName="payload-processing" containerID="cri-o://d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4" gracePeriod=30 Jul 29 05:01:02.204449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.204412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:02.292876 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.292847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:02.357358 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.357305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:02.421297 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.421273 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 05:01:02.529056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.529028 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6xk6\" (UniqueName: \"kubernetes.io/projected/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-kube-api-access-j6xk6\") pod \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " Jul 29 05:01:02.529252 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.529065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-plugins-config-volume\") pod \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\" (UID: \"66e27bea-5e1f-4045-9a4c-2b4e44d3428c\") " Jul 29 05:01:02.529468 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.529444 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66e27bea-5e1f-4045-9a4c-2b4e44d3428c" (UID: "66e27bea-5e1f-4045-9a4c-2b4e44d3428c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:01:02.531266 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.531246 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-kube-api-access-j6xk6" (OuterVolumeSpecName: "kube-api-access-j6xk6") pod "66e27bea-5e1f-4045-9a4c-2b4e44d3428c" (UID: "66e27bea-5e1f-4045-9a4c-2b4e44d3428c"). InnerVolumeSpecName "kube-api-access-j6xk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:01:02.630200 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.630174 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6xk6\" (UniqueName: \"kubernetes.io/projected/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-kube-api-access-j6xk6\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:02.630200 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.630199 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e27bea-5e1f-4045-9a4c-2b4e44d3428c-plugins-config-volume\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:02.656079 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.656012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:02.820555 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.820525 2576 generic.go:356] "Generic (PLEG): container finished" podID="66e27bea-5e1f-4045-9a4c-2b4e44d3428c" containerID="d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4" exitCode=0 Jul 29 05:01:02.820719 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.820576 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" Jul 29 05:01:02.820719 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.820612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" event={"ID":"66e27bea-5e1f-4045-9a4c-2b4e44d3428c","Type":"ContainerDied","Data":"d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4"} Jul 29 05:01:02.820719 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.820664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws" event={"ID":"66e27bea-5e1f-4045-9a4c-2b4e44d3428c","Type":"ContainerDied","Data":"93518c659f13d3fb2e6cb36338107b5f49148e114fce99b17c925ba84d517501"} Jul 29 05:01:02.820719 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.820699 2576 scope.go:117] "RemoveContainer" containerID="d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4" Jul 29 05:01:02.830663 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.830646 2576 scope.go:117] "RemoveContainer" containerID="d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4" Jul 29 05:01:02.830945 ip-10-0-140-47 kubenswrapper[2576]: E0729 05:01:02.830925 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4\": container with ID starting with d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4 not found: ID does not exist" containerID="d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4" Jul 29 05:01:02.831023 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.830954 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4"} err="failed to get container status \"d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4\": rpc error: code = NotFound desc = could not find container \"d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4\": container with ID starting with d79c7d1248e6abd1732f576bd90e5d490ad1acb1de56209547cb9c286a2966f4 not found: ID does not exist" Jul 29 05:01:02.843345 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.843322 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws"] Jul 29 05:01:02.847343 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:02.847321 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2cb3bf-6fd8c5f4d-kprws"] Jul 29 05:01:04.196933 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:04.196877 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e27bea-5e1f-4045-9a4c-2b4e44d3428c" path="/var/lib/kubelet/pods/66e27bea-5e1f-4045-9a4c-2b4e44d3428c/volumes" Jul 29 05:01:08.297324 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:08.297281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:08.359226 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:08.359191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:08.548917 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:08.548818 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb"] Jul 29 05:01:08.549078 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:08.549055 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" containerID="cri-o://e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1" gracePeriod=30 Jul 29 05:01:08.675221 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:08.675188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:08.786050 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:08.786021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:13.814496 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.814432 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 05:01:13.885019 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.884986 2576 generic.go:356] "Generic (PLEG): container finished" podID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerID="e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1" exitCode=0 Jul 29 05:01:13.885176 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.885033 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" Jul 29 05:01:13.885176 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.885046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" event={"ID":"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd","Type":"ContainerDied","Data":"e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1"} Jul 29 05:01:13.885176 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.885121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb" event={"ID":"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd","Type":"ContainerDied","Data":"875d8b2b11be7b19ac895adefceb1cc53ada0ae21e5a4f7897c395dcc4dcd113"} Jul 29 05:01:13.885176 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.885144 2576 scope.go:117] "RemoveContainer" containerID="e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1" Jul 29 05:01:13.894572 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.894557 2576 scope.go:117] "RemoveContainer" containerID="6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795" Jul 29 05:01:13.903043 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.903027 2576 scope.go:117] "RemoveContainer" containerID="e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1" Jul 29 05:01:13.903297 ip-10-0-140-47 kubenswrapper[2576]: E0729 05:01:13.903280 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1\": container with ID starting with e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1 not found: ID does not exist" containerID="e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1" Jul 29 05:01:13.903348 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.903306 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1"} err="failed to get container status \"e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1\": rpc error: code = NotFound desc = could not find container \"e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1\": container with ID starting with e21f3b1b5d56b6b16f50a75b822da5071f7a9109a0d4240079b677447f2ed3a1 not found: ID does not exist" Jul 29 05:01:13.903348 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.903323 2576 scope.go:117] "RemoveContainer" containerID="6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795" Jul 29 05:01:13.903542 ip-10-0-140-47 kubenswrapper[2576]: E0729 05:01:13.903525 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795\": container with ID starting with 6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795 not found: ID does not exist" containerID="6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795" Jul 29 05:01:13.903583 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.903555 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795"} err="failed to get container status \"6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795\": rpc error: code = NotFound desc = could not find container \"6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795\": container with ID starting with 6df71b782ebf32e6f28e4736860e037df537d41e6cf4cc1dc12adc82a39e6795 not found: ID does not exist" Jul 29 05:01:13.927426 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927404 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-podinfo\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927510 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927429 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istiod-ca-cert\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927510 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-credential-socket\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927608 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927590 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-envoy\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927658 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927634 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-token\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927712 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927694 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:01:13.927712 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927697 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sb6t\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-kube-api-access-8sb6t\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927822 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-certs\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927822 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927763 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-data\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.927822 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927794 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:01:13.928046 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.927822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-socket\") pod \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\" (UID: \"6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd\") " Jul 29 05:01:13.928101 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.928054 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:01:13.928212 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.928191 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istiod-ca-cert\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:13.928565 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.928533 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-credential-socket\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:13.928565 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.928236 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:01:13.928711 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.928559 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-certs\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:13.928771 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.928712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-data" (OuterVolumeSpecName: "istio-data") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:01:13.929940 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.929907 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:01:13.929940 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.929921 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-kube-api-access-8sb6t" (OuterVolumeSpecName: "kube-api-access-8sb6t") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "kube-api-access-8sb6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:01:13.930292 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.930274 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 05:01:13.930341 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:13.930309 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-token" (OuterVolumeSpecName: "istio-token") pod "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" (UID: "6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:01:14.029493 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.029463 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-data\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:14.029493 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.029488 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-workload-socket\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:14.029493 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.029499 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-podinfo\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:14.029707 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.029508 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-envoy\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:14.029707 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.029515 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-istio-token\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:14.029707 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.029523 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sb6t\" (UniqueName: \"kubernetes.io/projected/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd-kube-api-access-8sb6t\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:14.207587 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.207560 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb"] Jul 29 05:01:14.211065 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:14.211041 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2cb3bf-openshift-default-5496d94c56-888rb"] Jul 29 05:01:16.197618 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:16.197580 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" path="/var/lib/kubelet/pods/6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd/volumes" Jul 29 05:01:16.246457 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:16.246420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:19.462323 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:19.462286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:19.532246 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:19.532213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:19.587300 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:19.587268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:19.644047 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:19.643940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:20.003023 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:20.002985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:26.111459 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:26.111419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:26.167578 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:26.167546 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:26.359991 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:26.359959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:36.858696 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:36.858662 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6"] Jul 29 05:01:36.859200 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:36.859066 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" podUID="48447314-066a-4e17-a92d-53cabf5487b5" containerName="maas-api" containerID="cri-o://cbe4ded54a0bebcde25701bdaccabfbfc568ce477fcda5dce015d631011e1162" gracePeriod=30 Jul 29 05:01:36.891601 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:36.891568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:36.928955 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:36.928926 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp"] Jul 29 05:01:36.929192 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:36.929163 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" podUID="d7885895-2bc7-459e-8b51-9bc0b290aea3" containerName="payload-processing" containerID="cri-o://2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc" gracePeriod=30 Jul 29 05:01:36.965362 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:36.965336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:37.016473 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.016447 2576 generic.go:356] "Generic (PLEG): container finished" podID="48447314-066a-4e17-a92d-53cabf5487b5" containerID="cbe4ded54a0bebcde25701bdaccabfbfc568ce477fcda5dce015d631011e1162" exitCode=0 Jul 29 05:01:37.016612 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.016526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" event={"ID":"48447314-066a-4e17-a92d-53cabf5487b5","Type":"ContainerDied","Data":"cbe4ded54a0bebcde25701bdaccabfbfc568ce477fcda5dce015d631011e1162"} Jul 29 05:01:37.065557 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.065527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:37.204221 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.204198 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 05:01:37.207240 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.207212 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 05:01:37.331728 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.331700 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2snt\" (UniqueName: \"kubernetes.io/projected/d7885895-2bc7-459e-8b51-9bc0b290aea3-kube-api-access-k2snt\") pod \"d7885895-2bc7-459e-8b51-9bc0b290aea3\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " Jul 29 05:01:37.331937 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.331743 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48447314-066a-4e17-a92d-53cabf5487b5-openshift-service-ca\") pod \"48447314-066a-4e17-a92d-53cabf5487b5\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " Jul 29 05:01:37.331937 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.331785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls\") pod \"48447314-066a-4e17-a92d-53cabf5487b5\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " Jul 29 05:01:37.331937 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.331815 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7885895-2bc7-459e-8b51-9bc0b290aea3-plugins-config-volume\") pod \"d7885895-2bc7-459e-8b51-9bc0b290aea3\" (UID: \"d7885895-2bc7-459e-8b51-9bc0b290aea3\") " Jul 29 05:01:37.332135 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.331972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqwrn\" (UniqueName: \"kubernetes.io/projected/48447314-066a-4e17-a92d-53cabf5487b5-kube-api-access-zqwrn\") pod \"48447314-066a-4e17-a92d-53cabf5487b5\" (UID: \"48447314-066a-4e17-a92d-53cabf5487b5\") " Jul 29 05:01:37.332298 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.332271 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7885895-2bc7-459e-8b51-9bc0b290aea3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7885895-2bc7-459e-8b51-9bc0b290aea3" (UID: "d7885895-2bc7-459e-8b51-9bc0b290aea3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:01:37.332392 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.332314 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48447314-066a-4e17-a92d-53cabf5487b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48447314-066a-4e17-a92d-53cabf5487b5" (UID: "48447314-066a-4e17-a92d-53cabf5487b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:01:37.332977 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.332925 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48447314-066a-4e17-a92d-53cabf5487b5-openshift-service-ca\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:37.332977 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.332950 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7885895-2bc7-459e-8b51-9bc0b290aea3-plugins-config-volume\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:37.334432 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.334397 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48447314-066a-4e17-a92d-53cabf5487b5-kube-api-access-zqwrn" (OuterVolumeSpecName: "kube-api-access-zqwrn") pod "48447314-066a-4e17-a92d-53cabf5487b5" (UID: "48447314-066a-4e17-a92d-53cabf5487b5"). InnerVolumeSpecName "kube-api-access-zqwrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:01:37.334569 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.334549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48447314-066a-4e17-a92d-53cabf5487b5" (UID: "48447314-066a-4e17-a92d-53cabf5487b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 05:01:37.334629 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.334565 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7885895-2bc7-459e-8b51-9bc0b290aea3-kube-api-access-k2snt" (OuterVolumeSpecName: "kube-api-access-k2snt") pod "d7885895-2bc7-459e-8b51-9bc0b290aea3" (UID: "d7885895-2bc7-459e-8b51-9bc0b290aea3"). InnerVolumeSpecName "kube-api-access-k2snt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:01:37.433466 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.433445 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2snt\" (UniqueName: \"kubernetes.io/projected/d7885895-2bc7-459e-8b51-9bc0b290aea3-kube-api-access-k2snt\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:37.433466 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.433468 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48447314-066a-4e17-a92d-53cabf5487b5-maas-api-tls\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:37.433617 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.433477 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqwrn\" (UniqueName: \"kubernetes.io/projected/48447314-066a-4e17-a92d-53cabf5487b5-kube-api-access-zqwrn\") on node \"ip-10-0-140-47.ec2.internal\" DevicePath \"\"" Jul 29 05:01:37.915602 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:37.915575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:38.023966 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.023932 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7885895-2bc7-459e-8b51-9bc0b290aea3" containerID="2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc" exitCode=0 Jul 29 05:01:38.024158 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.023987 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" Jul 29 05:01:38.024158 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.024018 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" event={"ID":"d7885895-2bc7-459e-8b51-9bc0b290aea3","Type":"ContainerDied","Data":"2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc"} Jul 29 05:01:38.024158 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.024071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp" event={"ID":"d7885895-2bc7-459e-8b51-9bc0b290aea3","Type":"ContainerDied","Data":"3e172b57ad379617944335970df7edcbf34aa397c9d1f58b7bcfd527df3b7b07"} Jul 29 05:01:38.024158 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.024103 2576 scope.go:117] "RemoveContainer" containerID="2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc" Jul 29 05:01:38.025717 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.025676 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" Jul 29 05:01:38.025842 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.025699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6" event={"ID":"48447314-066a-4e17-a92d-53cabf5487b5","Type":"ContainerDied","Data":"bd2b17dc9e59ade50d31977edf664bb946e6125e8af0ec92ae47d6c4dfe1e160"} Jul 29 05:01:38.036237 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.036091 2576 scope.go:117] "RemoveContainer" containerID="2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc" Jul 29 05:01:38.036583 ip-10-0-140-47 kubenswrapper[2576]: E0729 05:01:38.036496 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc\": container with ID starting with 2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc not found: ID does not exist" containerID="2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc" Jul 29 05:01:38.036583 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.036536 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc"} err="failed to get container status \"2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc\": rpc error: code = NotFound desc = could not find container \"2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc\": container with ID starting with 2a2284d38055fce693587958762034872625f214cca59f5a80e8ace9db1310fc not found: ID does not exist" Jul 29 05:01:38.036583 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.036560 2576 scope.go:117] "RemoveContainer" containerID="cbe4ded54a0bebcde25701bdaccabfbfc568ce477fcda5dce015d631011e1162" Jul 29 05:01:38.054250 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.054221 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6"] Jul 29 05:01:38.062795 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.062769 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b550f6-5569554b5c-dtgf6"] Jul 29 05:01:38.076782 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.076757 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp"] Jul 29 05:01:38.080722 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.080698 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b550f6-c95fd59f8-99nbp"] Jul 29 05:01:38.198017 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.197940 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48447314-066a-4e17-a92d-53cabf5487b5" path="/var/lib/kubelet/pods/48447314-066a-4e17-a92d-53cabf5487b5/volumes" Jul 29 05:01:38.198315 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:38.198302 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7885895-2bc7-459e-8b51-9bc0b290aea3" path="/var/lib/kubelet/pods/d7885895-2bc7-459e-8b51-9bc0b290aea3/volumes" Jul 29 05:01:40.511076 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:40.511045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:43.392130 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:43.392100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:43.447025 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:43.446997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:44.749075 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:44.749043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:44.800188 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:44.800157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:01:49.486735 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:49.486703 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5hjg5_ccf78918-f12e-43bc-b98d-8c09acab26ae/manager/0.log" Jul 29 05:01:49.604378 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:49.604344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q9xfg_58fdab89-2cb1-4b21-a69e-4510fa300b04/manager/0.log" Jul 29 05:01:49.828766 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:49.828682 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v8ksb_ad01a230-dc80-4b63-946c-df141ae371f6/server/0.log" Jul 29 05:01:49.939373 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:49.939342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h9sc8_f981cf48-7ab5-4596-80a4-cd5910b8f5c1/manager/0.log" Jul 29 05:01:50.043550 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:50.043526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-fbcdv_edc39eb8-388c-4fce-a9da-3281b04150c7/copy-manifests/0.log" Jul 29 05:01:50.055390 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:50.055359 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-fbcdv_edc39eb8-388c-4fce-a9da-3281b04150c7/manager/0.log" Jul 29 05:01:50.623708 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:50.623685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74b74d4d58-qs8rn_a703b6a4-59d0-4e04-ba12-b738620bacce/maas-api/0.log" Jul 29 05:01:51.044723 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:51.044643 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vwhnq_25caae94-4fd5-4a86-9f79-4730a99c1cea/postgres/0.log" Jul 29 05:01:51.793119 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:51.793089 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v_05790ad2-5d86-4ebd-b90c-83c79f96ff07/util/0.log" Jul 29 05:01:51.801296 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:51.801271 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v_05790ad2-5d86-4ebd-b90c-83c79f96ff07/pull/0.log" Jul 29 05:01:51.810818 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:51.810795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v_05790ad2-5d86-4ebd-b90c-83c79f96ff07/extract/0.log" Jul 29 05:01:51.922943 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:51.922914 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z_257ae2f6-4912-4362-b500-b886e8a3c5be/util/0.log" Jul 29 05:01:51.931948 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:51.931922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z_257ae2f6-4912-4362-b500-b886e8a3c5be/pull/0.log" Jul 29 05:01:51.937779 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:51.937759 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z_257ae2f6-4912-4362-b500-b886e8a3c5be/extract/0.log" Jul 29 05:01:52.044975 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:52.044871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf_3a214469-3308-42d4-b967-9d9b36e96b3b/util/0.log" Jul 29 05:01:52.053927 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:52.053911 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf_3a214469-3308-42d4-b967-9d9b36e96b3b/pull/0.log" Jul 29 05:01:52.061822 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:52.061806 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf_3a214469-3308-42d4-b967-9d9b36e96b3b/extract/0.log" Jul 29 05:01:52.173057 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:52.173030 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw_399cdbd9-8aa0-4f0d-8b29-191573cc1699/pull/0.log" Jul 29 05:01:52.180781 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:52.180766 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw_399cdbd9-8aa0-4f0d-8b29-191573cc1699/extract/0.log" Jul 29 05:01:52.192580 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:52.192560 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw_399cdbd9-8aa0-4f0d-8b29-191573cc1699/util/0.log" Jul 29 05:01:52.536345 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:52.536319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2ts5t_5c99abd3-fb23-4110-b208-7b78f0fe89ff/manager/0.log" Jul 29 05:01:53.115328 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:53.115294 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ztwbz_a966ac0e-eeb3-41a8-bf97-fc98c424c8c8/manager/0.log" Jul 29 05:01:53.575738 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:53.575709 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b69pt_763a30c4-5558-486d-9563-e6def2625bfa/discovery/0.log" Jul 29 05:01:53.683728 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:53.683699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-8r9lt_73085c0c-0c12-49e1-b064-17b725ed001e/kube-auth-proxy/0.log" Jul 29 05:01:54.256747 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:54.256722 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f89b64c-jk642_28ed722e-2b3c-4d7e-90ed-5f247f1b0f46/router/0.log" Jul 29 05:01:54.586907 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:54.586812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bzhcx_173252c7-dcf5-420a-8575-3ecaa091e668/main/0.log" Jul 29 05:01:54.699377 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:54.699347 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-flcm2_680f2c4f-1b87-4e9a-af31-15ec2a1cc5fd/main/0.log" Jul 29 05:01:55.027426 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:01:55.027395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4ztxz_e0f696de-2366-4b75-8787-d7c3114a8a13/main/0.log" Jul 29 05:02:00.010811 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:00.010778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:02:00.457779 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:00.457710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:02:00.543227 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:00.543198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" limit=1024 Jul 29 05:02:03.497254 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:03.497224 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x6mqc_525a2c3e-cca1-435f-a3d6-7b94f465d5f5/global-pull-secret-syncer/0.log" Jul 29 05:02:03.550354 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:03.550328 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6jmqc_12b19858-6ae7-4c40-b65a-209e52534efd/konnectivity-agent/0.log" Jul 29 05:02:03.688308 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:03.688269 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-47.ec2.internal_b753081a1bef3704e4855e1302d0c324/haproxy/0.log" Jul 29 05:02:08.242058 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.242008 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v_05790ad2-5d86-4ebd-b90c-83c79f96ff07/extract/0.log" Jul 29 05:02:08.266847 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.266820 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v_05790ad2-5d86-4ebd-b90c-83c79f96ff07/util/0.log" Jul 29 05:02:08.291861 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.291782 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928d9v_05790ad2-5d86-4ebd-b90c-83c79f96ff07/pull/0.log" Jul 29 05:02:08.318163 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.318141 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z_257ae2f6-4912-4362-b500-b886e8a3c5be/extract/0.log" Jul 29 05:02:08.339556 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.339532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z_257ae2f6-4912-4362-b500-b886e8a3c5be/util/0.log" Jul 29 05:02:08.364623 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.364606 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04h45z_257ae2f6-4912-4362-b500-b886e8a3c5be/pull/0.log" Jul 29 05:02:08.393497 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.393474 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf_3a214469-3308-42d4-b967-9d9b36e96b3b/extract/0.log" Jul 29 05:02:08.419251 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.419232 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf_3a214469-3308-42d4-b967-9d9b36e96b3b/util/0.log" Jul 29 05:02:08.440304 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.440282 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7396zzf_3a214469-3308-42d4-b967-9d9b36e96b3b/pull/0.log" Jul 29 05:02:08.467040 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.467018 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw_399cdbd9-8aa0-4f0d-8b29-191573cc1699/extract/0.log" Jul 29 05:02:08.489519 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.489500 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw_399cdbd9-8aa0-4f0d-8b29-191573cc1699/util/0.log" Jul 29 05:02:08.511625 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:08.511610 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qgfnw_399cdbd9-8aa0-4f0d-8b29-191573cc1699/pull/0.log" Jul 29 05:02:09.203169 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:09.203143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2ts5t_5c99abd3-fb23-4110-b208-7b78f0fe89ff/manager/0.log" Jul 29 05:02:09.461317 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:09.461234 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ztwbz_a966ac0e-eeb3-41a8-bf97-fc98c424c8c8/manager/0.log" Jul 29 05:02:11.296297 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:11.296268 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-f9gqc_fae15b77-656e-4080-a60b-ced44902b861/cluster-monitoring-operator/0.log" Jul 29 05:02:11.469245 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:11.469220 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cjng5_cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f/node-exporter/0.log" Jul 29 05:02:11.495186 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:11.495159 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cjng5_cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f/kube-rbac-proxy/0.log" Jul 29 05:02:11.516764 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:11.516744 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cjng5_cf4df7e1-0c63-4ffa-aa1a-9cffd8ba9c1f/init-textfile/0.log" Jul 29 05:02:12.074913 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:12.074868 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c47c9867f-g6fvr_98231da0-b5d1-473a-9721-f43f769935ae/telemeter-client/0.log" Jul 29 05:02:12.096105 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:12.096071 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c47c9867f-g6fvr_98231da0-b5d1-473a-9721-f43f769935ae/reload/0.log" Jul 29 05:02:12.137417 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:12.137376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c47c9867f-g6fvr_98231da0-b5d1-473a-9721-f43f769935ae/kube-rbac-proxy/0.log" Jul 29 05:02:13.366358 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366323 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg"] Jul 29 05:02:13.366803 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366785 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e27bea-5e1f-4045-9a4c-2b4e44d3428c" containerName="payload-processing" Jul 29 05:02:13.366851 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366805 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e27bea-5e1f-4045-9a4c-2b4e44d3428c" containerName="payload-processing" Jul 29 05:02:13.366851 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366818 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" Jul 29 05:02:13.366851 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366825 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" Jul 29 05:02:13.366851 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366835 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49299e45-4572-40fb-a505-3f301ffe5491" containerName="payload-processing" Jul 29 05:02:13.366851 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366840 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49299e45-4572-40fb-a505-3f301ffe5491" containerName="payload-processing" Jul 29 05:02:13.366851 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366849 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718443cc-e97c-4841-83ac-187a847f232c" containerName="maas-api" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366854 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="718443cc-e97c-4841-83ac-187a847f232c" containerName="maas-api" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366865 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7885895-2bc7-459e-8b51-9bc0b290aea3" containerName="payload-processing" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366871 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7885895-2bc7-459e-8b51-9bc0b290aea3" containerName="payload-processing" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366876 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48447314-066a-4e17-a92d-53cabf5487b5" containerName="maas-api" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366896 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="48447314-066a-4e17-a92d-53cabf5487b5" containerName="maas-api" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366903 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366907 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366979 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="48447314-066a-4e17-a92d-53cabf5487b5" containerName="maas-api" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366987 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49299e45-4572-40fb-a505-3f301ffe5491" containerName="payload-processing" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366994 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7885895-2bc7-459e-8b51-9bc0b290aea3" containerName="payload-processing" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.366999 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e27bea-5e1f-4045-9a4c-2b4e44d3428c" containerName="payload-processing" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.367007 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.367012 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6462b5a9-80a3-41eb-bc52-0cbfe7c64cdd" containerName="istio-proxy" Jul 29 05:02:13.367056 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.367018 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="718443cc-e97c-4841-83ac-187a847f232c" containerName="maas-api" Jul 29 05:02:13.369581 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.369565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.372274 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.372256 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2sff\"/\"openshift-service-ca.crt\"" Jul 29 05:02:13.373127 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.373114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2sff\"/\"kube-root-ca.crt\"" Jul 29 05:02:13.373199 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.373131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q2sff\"/\"default-dockercfg-xt2ns\"" Jul 29 05:02:13.377064 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.377040 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg"] Jul 29 05:02:13.442039 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.442011 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-bj6qs_a719ded6-afb9-4ac3-8c4f-1fea94627815/networking-console-plugin/0.log" Jul 29 05:02:13.451065 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.451043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-podres\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.451204 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.451132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-proc\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.451204 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.451154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-sys\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.451204 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.451178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-lib-modules\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.451328 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.451205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8lss\" (UniqueName: \"kubernetes.io/projected/7a7e4632-5c49-47a3-9cac-b0e08e32c507-kube-api-access-s8lss\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552211 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-proc\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552211 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-sys\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-lib-modules\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8lss\" (UniqueName: \"kubernetes.io/projected/7a7e4632-5c49-47a3-9cac-b0e08e32c507-kube-api-access-s8lss\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-podres\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-proc\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-sys\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-lib-modules\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.552449 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.552435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a7e4632-5c49-47a3-9cac-b0e08e32c507-podres\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.560997 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.560969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8lss\" (UniqueName: \"kubernetes.io/projected/7a7e4632-5c49-47a3-9cac-b0e08e32c507-kube-api-access-s8lss\") pod \"perf-node-gather-daemonset-777pg\" (UID: \"7a7e4632-5c49-47a3-9cac-b0e08e32c507\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.680946 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.680912 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:13.819716 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:13.819690 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg"] Jul 29 05:02:13.821643 ip-10-0-140-47 kubenswrapper[2576]: W0729 05:02:13.821619 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a7e4632_5c49_47a3_9cac_b0e08e32c507.slice/crio-74d5389ddc17322b02d06edd62313ebf233b9ae337042d4a760dad0d002e8255 WatchSource:0}: Error finding container 74d5389ddc17322b02d06edd62313ebf233b9ae337042d4a760dad0d002e8255: Status 404 returned error can't find the container with id 74d5389ddc17322b02d06edd62313ebf233b9ae337042d4a760dad0d002e8255 Jul 29 05:02:14.092380 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:14.092300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/1.log" Jul 29 05:02:14.098263 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:14.098245 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kqvg8_db6c8ff1-edf0-406d-8438-e502e0a656c7/console-operator/2.log" Jul 29 05:02:14.232450 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:14.232421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" event={"ID":"7a7e4632-5c49-47a3-9cac-b0e08e32c507","Type":"ContainerStarted","Data":"bec2b43a36f90beb23a5bb06d5e418a42c8bc81bdfcbc83b4a9ccb507329af34"} Jul 29 05:02:14.232598 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:14.232474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" event={"ID":"7a7e4632-5c49-47a3-9cac-b0e08e32c507","Type":"ContainerStarted","Data":"74d5389ddc17322b02d06edd62313ebf233b9ae337042d4a760dad0d002e8255"} Jul 29 05:02:14.232598 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:14.232505 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:14.248930 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:14.248867 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" podStartSLOduration=1.248852846 podStartE2EDuration="1.248852846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 05:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:02:14.246351752 +0000 UTC m=+4016.538448111" watchObservedRunningTime="2026-07-29 05:02:14.248852846 +0000 UTC m=+4016.540949207" Jul 29 05:02:14.631561 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:14.631534 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-n2dwp_9a404653-f311-4db0-8c78-10e4e9d73000/download-server/0.log" Jul 29 05:02:15.200946 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:15.200919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-mc8zb_525d31bf-6341-4d2d-b238-3313395d1e28/volume-data-source-validator/0.log" Jul 29 05:02:16.209251 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:16.209222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2lxw_fd9e5cff-51ba-49af-9156-5d6714453eed/dns/0.log" Jul 29 05:02:16.231403 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:16.231384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2lxw_fd9e5cff-51ba-49af-9156-5d6714453eed/kube-rbac-proxy/0.log" Jul 29 05:02:16.338061 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:16.338036 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd8bl_0a7006fc-a2b3-4ff8-8126-e6e46a26fa7b/dns-node-resolver/0.log" Jul 29 05:02:16.942483 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:16.942451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnhv8_e1d96163-4e1b-4d7f-ad07-55d7488acb5e/node-ca/0.log" Jul 29 05:02:17.988171 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:17.988144 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b69pt_763a30c4-5558-486d-9563-e6def2625bfa/discovery/0.log" Jul 29 05:02:18.009851 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:18.009827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-8r9lt_73085c0c-0c12-49e1-b064-17b725ed001e/kube-auth-proxy/0.log" Jul 29 05:02:18.202682 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:18.202646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f89b64c-jk642_28ed722e-2b3c-4d7e-90ed-5f247f1b0f46/router/0.log" Jul 29 05:02:18.968048 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:18.968019 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmqfs_bdbb1392-e8d1-4fdf-9dcb-4a3c689d8205/serve-healthcheck-canary/0.log" Jul 29 05:02:19.489419 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:19.489395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rhf5t_3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49/insights-operator/0.log" Jul 29 05:02:19.490216 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:19.490202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rhf5t_3a9712b1-dd0e-4177-a5c0-4f2a8a5d1e49/insights-operator/1.log" Jul 29 05:02:19.576712 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:19.576664 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9c4g7_12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7/kube-rbac-proxy/0.log" Jul 29 05:02:19.600995 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:19.600973 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9c4g7_12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7/exporter/0.log" Jul 29 05:02:19.621748 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:19.621731 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9c4g7_12b6a5a8-bb22-47f5-8018-0e15f4ca4bb7/extractor/0.log" Jul 29 05:02:20.248294 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:20.248268 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-777pg" Jul 29 05:02:21.860962 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:21.860935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5hjg5_ccf78918-f12e-43bc-b98d-8c09acab26ae/manager/0.log" Jul 29 05:02:22.090623 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:22.090583 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q9xfg_58fdab89-2cb1-4b21-a69e-4510fa300b04/manager/0.log" Jul 29 05:02:22.346447 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:22.346420 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v8ksb_ad01a230-dc80-4b63-946c-df141ae371f6/server/0.log" Jul 29 05:02:22.421090 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:22.421065 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h9sc8_f981cf48-7ab5-4596-80a4-cd5910b8f5c1/manager/0.log" Jul 29 05:02:22.507157 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:22.507130 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-fbcdv_edc39eb8-388c-4fce-a9da-3281b04150c7/manager/0.log" Jul 29 05:02:22.527073 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:22.527054 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-fbcdv_edc39eb8-388c-4fce-a9da-3281b04150c7/copy-manifests/0.log" Jul 29 05:02:24.637675 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:24.637647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b69pt_763a30c4-5558-486d-9563-e6def2625bfa/discovery/0.log" Jul 29 05:02:24.649353 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:24.649329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-8r9lt_73085c0c-0c12-49e1-b064-17b725ed001e/kube-auth-proxy/0.log" Jul 29 05:02:24.789048 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:24.789018 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f89b64c-jk642_28ed722e-2b3c-4d7e-90ed-5f247f1b0f46/router/0.log" Jul 29 05:02:26.255503 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:26.255472 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b69pt_763a30c4-5558-486d-9563-e6def2625bfa/discovery/0.log" Jul 29 05:02:26.268537 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:26.268511 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-8r9lt_73085c0c-0c12-49e1-b064-17b725ed001e/kube-auth-proxy/0.log" Jul 29 05:02:26.411344 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:26.411314 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f89b64c-jk642_28ed722e-2b3c-4d7e-90ed-5f247f1b0f46/router/0.log" Jul 29 05:02:29.440321 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:29.440286 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl_596212fb-ef59-4d1a-9770-86fce25ea36b/kube-storage-version-migrator-operator/1.log" Jul 29 05:02:29.441611 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:29.441340 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9tkbl_596212fb-ef59-4d1a-9770-86fce25ea36b/kube-storage-version-migrator-operator/0.log" Jul 29 05:02:30.451297 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.451263 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7mv9l_4d5643a1-ea0e-4024-9295-d295ca13a9f1/kube-multus/0.log" Jul 29 05:02:30.624394 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.624358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gt4v_4b079719-cf88-4c8b-98fe-0f72beec6c97/kube-multus-additional-cni-plugins/0.log" Jul 29 05:02:30.643690 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.643661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gt4v_4b079719-cf88-4c8b-98fe-0f72beec6c97/egress-router-binary-copy/0.log" Jul 29 05:02:30.663633 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.663613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gt4v_4b079719-cf88-4c8b-98fe-0f72beec6c97/cni-plugins/0.log" Jul 29 05:02:30.684941 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.684923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gt4v_4b079719-cf88-4c8b-98fe-0f72beec6c97/bond-cni-plugin/0.log" Jul 29 05:02:30.707537 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.707484 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gt4v_4b079719-cf88-4c8b-98fe-0f72beec6c97/routeoverride-cni/0.log" Jul 29 05:02:30.727581 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.727559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gt4v_4b079719-cf88-4c8b-98fe-0f72beec6c97/whereabouts-cni-bincopy/0.log" Jul 29 05:02:30.749169 ip-10-0-140-47 kubenswrapper[2576]: I0729 05:02:30.749147 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gt4v_4b079719-cf88-4c8b-98fe-0f72beec6c97/whereabouts-cni/0.log"