Jul 24 19:26:08.133409 ip-10-0-141-238 systemd[1]: Starting Kubernetes Kubelet... Jul 24 19:26:08.574021 ip-10-0-141-238 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 19:26:08.574021 ip-10-0-141-238 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 19:26:08.574021 ip-10-0-141-238 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 19:26:08.574021 ip-10-0-141-238 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 19:26:08.574021 ip-10-0-141-238 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 19:26:08.575134 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.575049 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 19:26:08.580440 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580419 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 19:26:08.580440 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580436 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 19:26:08.580440 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580440 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 19:26:08.580440 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580443 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 19:26:08.580440 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580446 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580450 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580454 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580457 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580460 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580463 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580465 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580468 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580471 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580474 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580477 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580480 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580484 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580487 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580489 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580492 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580495 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580498 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580501 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580503 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 19:26:08.580629 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580506 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580509 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580511 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580514 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580523 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580526 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580528 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580531 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580533 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580536 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580539 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580541 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580544 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580546 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580550 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580553 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580556 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580558 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580561 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580564 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 19:26:08.581165 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580567 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580569 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580572 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580574 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580577 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580579 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580582 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580584 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580587 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580589 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580592 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580594 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580597 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580599 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580602 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580605 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580607 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580610 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580613 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580616 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 19:26:08.581722 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580618 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580622 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580625 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580630 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580635 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580639 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580642 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580646 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580648 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580651 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580653 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580656 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580659 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580661 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580664 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580666 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580669 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580671 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580674 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 19:26:08.582224 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580683 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580685 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580688 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580690 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580692 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580695 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580697 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580700 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580702 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580704 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580707 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580710 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580714 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580717 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580720 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.580722 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581410 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581418 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581422 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 19:26:08.582690 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581426 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581429 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581432 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581435 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581438 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581441 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581444 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581446 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581449 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581451 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581454 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581456 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581459 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581461 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581464 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581467 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581469 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581471 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581474 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581476 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 19:26:08.583137 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581479 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581481 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581483 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581487 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581491 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581493 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581496 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581498 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581501 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581503 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581506 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581508 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581511 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581514 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581516 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581520 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581522 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581525 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581527 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581530 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 19:26:08.583636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581532 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581536 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581540 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581542 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581545 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581548 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581551 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581554 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581556 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581559 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581561 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581564 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581566 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581569 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581572 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581575 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581578 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581581 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581584 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 19:26:08.584173 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581586 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581589 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581592 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581595 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581598 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581602 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581605 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581608 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581611 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581613 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581615 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581618 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581621 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581624 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581626 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581629 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581631 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581633 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581636 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581638 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 19:26:08.584645 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581641 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581643 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581646 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581648 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581651 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581653 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581655 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581658 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581660 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581663 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581665 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581668 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581670 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581672 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581675 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581677 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.581680 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582023 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582028 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582031 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 19:26:08.585170 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582033 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582036 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582039 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582041 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582046 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582049 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582052 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582055 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582057 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582060 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582063 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582065 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582068 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582071 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582074 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582077 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582079 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582081 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582084 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 19:26:08.585638 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582086 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582089 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582091 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582094 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582096 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582098 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582101 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582103 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582106 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582108 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582114 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582118 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582120 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582123 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582125 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582128 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582130 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582134 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582137 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 19:26:08.586083 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582140 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582142 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582145 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582147 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582163 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582166 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582169 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582172 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582175 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582177 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582180 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582182 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582185 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582187 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582191 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582195 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582198 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582201 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582204 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 19:26:08.586572 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582206 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582209 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582212 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582215 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582219 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582222 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582225 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582228 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582231 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582234 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582237 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582240 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582242 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582245 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582248 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582250 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582253 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582256 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582259 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582262 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 19:26:08.587028 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582264 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582266 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582269 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582272 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582274 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582277 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582279 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582282 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582284 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582286 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582289 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582292 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582294 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582297 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582299 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582302 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582306 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582308 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582311 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582378 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582385 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 19:26:08.587516 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582391 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582394 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582398 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582401 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582405 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582409 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582412 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582415 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582418 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582421 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582424 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582427 2570 flags.go:64] FLAG: --cgroup-root="" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582430 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582433 2570 flags.go:64] FLAG: --client-ca-file="" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582436 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582439 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582443 2570 flags.go:64] FLAG: --cluster-domain="" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582446 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582448 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582451 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582454 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582458 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582461 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 19:26:08.588076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582464 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582467 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582469 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582473 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582477 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582480 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582483 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582487 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582490 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582493 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582496 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582498 2570 flags.go:64] FLAG: --enable-server="true" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582501 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582506 2570 flags.go:64] FLAG: --event-burst="100" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582509 2570 flags.go:64] FLAG: --event-qps="50" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582512 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582515 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582518 2570 flags.go:64] FLAG: --eviction-hard="" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582521 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582524 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582527 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582530 2570 flags.go:64] FLAG: --eviction-soft="" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582533 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582535 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582539 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 19:26:08.588651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582541 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582544 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582547 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582549 2570 flags.go:64] FLAG: --feature-gates="" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582553 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582556 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582559 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582562 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582564 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582567 2570 flags.go:64] FLAG: --help="false" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582570 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-141-238.ec2.internal" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582575 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582578 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582581 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582584 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582587 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582590 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582593 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582596 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582598 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582601 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582604 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582607 2570 flags.go:64] FLAG: --kube-reserved="" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582610 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 19:26:08.589636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582612 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582615 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582618 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582621 2570 flags.go:64] FLAG: --lock-file="" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582623 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582626 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582629 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582634 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582640 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582643 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582645 2570 flags.go:64] FLAG: --logging-format="text" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582648 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582651 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582654 2570 flags.go:64] FLAG: --manifest-url="" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582657 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582660 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582663 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582667 2570 flags.go:64] FLAG: --max-pods="110" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582670 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582674 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582677 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582680 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582683 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582685 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582688 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 19:26:08.590658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582695 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582698 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582701 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582704 2570 flags.go:64] FLAG: --pod-cidr="" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582707 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582712 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582715 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582718 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582721 2570 flags.go:64] FLAG: --port="10250" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582723 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582726 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cd0d9ca9ae48e0f9" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582729 2570 flags.go:64] FLAG: --qos-reserved="" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582732 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582735 2570 flags.go:64] FLAG: --register-node="true" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582737 2570 flags.go:64] FLAG: --register-with-taints="" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582741 2570 flags.go:64] FLAG: --registry-burst="10" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582751 2570 flags.go:64] FLAG: --registry-qps="5" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582754 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582756 2570 flags.go:64] FLAG: --reserved-memory="" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582760 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582764 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582767 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582770 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582772 2570 flags.go:64] FLAG: --runonce="false" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582775 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 19:26:08.591454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582778 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582781 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582784 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582787 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582789 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582792 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582795 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582798 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582800 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582803 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582805 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582808 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582811 2570 flags.go:64] FLAG: --system-cgroups="" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582814 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582819 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582821 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582824 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582830 2570 flags.go:64] FLAG: --tls-min-version="" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582832 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582835 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582837 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582840 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582842 2570 flags.go:64] FLAG: --v="2" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582846 2570 flags.go:64] FLAG: --version="false" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582853 2570 flags.go:64] FLAG: --vmodule="" Jul 24 19:26:08.592285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582857 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.582860 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582982 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582987 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582990 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582994 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582997 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.582999 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583002 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583004 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583007 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583010 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583013 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583015 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583018 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583020 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583023 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583025 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583028 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583030 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 19:26:08.593362 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583033 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583035 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583038 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583040 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583043 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583045 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583048 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583050 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583053 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583055 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583057 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583060 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583063 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583066 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583071 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583074 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583077 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583079 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583082 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 19:26:08.593883 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583084 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583087 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583089 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583092 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583095 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583097 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583099 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583102 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583105 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583107 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583109 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583112 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583114 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583117 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583119 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583121 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583124 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583126 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583128 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 19:26:08.594585 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583131 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583162 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583166 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583169 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583172 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583175 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583177 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583180 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583184 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583187 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583189 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583192 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583194 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583196 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583199 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583202 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583205 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583207 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583210 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583212 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 19:26:08.595459 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583214 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583217 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583219 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583222 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583224 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583227 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583229 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583232 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583234 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583236 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583239 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583241 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583244 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583247 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583250 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583254 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583256 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583259 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583262 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 19:26:08.595976 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583265 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583275 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583277 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.583280 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.583285 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.591006 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.591024 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.591942 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.591956 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592040 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592049 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592055 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592059 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592064 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592069 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 19:26:08.596568 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592074 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592079 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592083 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592087 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592094 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592099 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592104 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592109 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592114 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592119 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592123 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592129 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592134 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592139 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592144 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592148 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592167 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592172 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592176 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 19:26:08.596961 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592181 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592185 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592189 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592193 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592200 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592204 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592209 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592213 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592218 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592222 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592229 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592235 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592240 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592244 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592249 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592254 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592259 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592263 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592268 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592272 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 19:26:08.597455 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592277 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592281 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592286 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592290 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592294 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592298 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592302 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592306 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592310 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592314 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592318 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592322 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592327 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592331 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592336 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592341 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592345 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592351 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592355 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592359 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 19:26:08.597925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592364 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592368 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592372 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592377 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592382 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592386 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592390 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592395 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592399 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592403 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592408 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592412 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592416 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592420 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592425 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592429 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592433 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592438 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592442 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592445 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 19:26:08.598464 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592450 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592454 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592459 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592463 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592467 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592471 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592476 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592480 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592485 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592491 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592498 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592503 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592507 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592511 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.592519 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 19:26:08.598982 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592693 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592702 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592707 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592713 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592717 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592721 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592725 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592732 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592737 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592742 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592748 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592754 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592758 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592763 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592767 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592772 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592776 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592780 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592784 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 19:26:08.599428 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592788 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592793 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592800 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592805 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592810 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592815 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592819 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592823 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592828 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592833 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592837 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592841 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592845 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592849 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592854 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592858 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592861 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592865 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592870 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592875 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 19:26:08.599925 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592879 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592883 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592887 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592891 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592895 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592899 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592903 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592907 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592911 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592915 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592919 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592923 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592927 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592931 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592935 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592952 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592956 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592960 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592964 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 19:26:08.600427 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592969 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592973 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592976 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592981 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592985 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592989 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592993 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.592997 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593001 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593005 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593008 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593012 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593016 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593020 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593024 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593028 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593032 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593036 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593040 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593044 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 19:26:08.601136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593048 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593052 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593056 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593060 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593063 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593067 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593071 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593075 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593079 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593083 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593088 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593092 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593096 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593100 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593104 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593108 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593112 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593116 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593120 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593124 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 19:26:08.601950 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:08.593128 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 19:26:08.602741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.593136 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 19:26:08.602741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.593286 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 19:26:08.602741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.596250 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 19:26:08.602741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.598024 2570 server.go:1019] "Starting client certificate rotation" Jul 24 19:26:08.602741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.598127 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 19:26:08.602741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.598206 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 19:26:08.602741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.601176 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 19:26:08.605020 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.604995 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 19:26:08.613596 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.613580 2570 log.go:25] "Validated CRI v1 runtime API" Jul 24 19:26:08.616997 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.616982 2570 log.go:25] "Validated CRI v1 image API" Jul 24 19:26:08.619349 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.619336 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 19:26:08.621970 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.621950 2570 fs.go:135] Filesystem UUIDs: map[06dab7da-2010-4680-85fc-5b86417e4e48:/dev/nvme0n1p3 0cbffec2-e491-4fcf-a076-8ae6371c0ad8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 24 19:26:08.622035 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.621968 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 19:26:08.626076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.626062 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 19:26:08.627961 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.627855 2570 manager.go:216] Machine: {Timestamp:2026-07-24 19:26:08.626466057 +0000 UTC m=+0.367270542 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3153919 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26538f5494f4878fe9b053b65f892e SystemUUID:ec26538f-5494-f487-8fe9-b053b65f892e BootID:9cdf9d66-0ef8-42ad-92c3-84864579aa92 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:65:b5:69:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:65:b5:69:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:23:ad:01:96:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 19:26:08.627961 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.627958 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 19:26:08.628061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.628033 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 19:26:08.631454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.631431 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 19:26:08.631580 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.631457 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-238.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 19:26:08.632294 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.632284 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 19:26:08.632328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.632296 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 19:26:08.632328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.632303 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 19:26:08.632328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.632317 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 19:26:08.633603 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.633594 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 19:26:08.636483 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.636473 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 24 19:26:08.636588 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.636580 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 19:26:08.636625 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.636609 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 24 19:26:08.636625 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.636618 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 19:26:08.636684 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.636628 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 19:26:08.636684 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.636637 2570 kubelet.go:398] "Adding apiserver pod source" Jul 24 19:26:08.636684 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.636646 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 19:26:08.637063 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.637051 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 19:26:08.637105 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.637071 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 19:26:08.640471 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.640336 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 19:26:08.642972 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.642956 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 19:26:08.643045 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.642985 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 19:26:08.645056 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645041 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 19:26:08.645056 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645058 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645064 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645070 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645078 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645083 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645091 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645096 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645102 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645107 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645115 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 19:26:08.645185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.645121 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 19:26:08.646073 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.646060 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 19:26:08.646073 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.646070 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 19:26:08.649020 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.648998 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 19:26:08.649102 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.649015 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 19:26:08.649102 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.649064 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-238.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 19:26:08.649270 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.649258 2570 server.go:1268] "Started kubelet" Jul 24 19:26:08.649368 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.649323 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 19:26:08.649423 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.649353 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 19:26:08.649423 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.649412 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 19:26:08.650169 ip-10-0-141-238 systemd[1]: Started Kubernetes Kubelet. Jul 24 19:26:08.650463 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.650439 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 24 19:26:08.651891 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.651878 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 19:26:08.655328 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.655310 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 19:26:08.655419 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.654239 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-238.ec2.internal.18c5505891699f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-238.ec2.internal,UID:ip-10-0-141-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-238.ec2.internal,},FirstTimestamp:2026-07-24 19:26:08.649232194 +0000 UTC m=+0.390036680,LastTimestamp:2026-07-24 19:26:08.649232194 +0000 UTC m=+0.390036680,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-238.ec2.internal,}" Jul 24 19:26:08.661368 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.661354 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 19:26:08.661441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.661370 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 19:26:08.661489 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.661447 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:08.662147 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662131 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 19:26:08.662147 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662133 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 19:26:08.662313 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662173 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 19:26:08.662364 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662345 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 24 19:26:08.662364 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662355 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 24 19:26:08.662601 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662589 2570 factory.go:153] Registering CRI-O factory Jul 24 19:26:08.662601 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662603 2570 factory.go:223] Registration of the crio container factory successfully Jul 24 19:26:08.662675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662640 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 19:26:08.662675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662646 2570 factory.go:55] Registering systemd factory Jul 24 19:26:08.662675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662651 2570 factory.go:223] Registration of the systemd container factory successfully Jul 24 19:26:08.662675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662666 2570 factory.go:103] Registering Raw factory Jul 24 19:26:08.662675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.662676 2570 manager.go:1196] Started watching for new ooms in manager Jul 24 19:26:08.663021 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.663010 2570 manager.go:349] Starting recovery of all containers Jul 24 19:26:08.672298 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.672270 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cc8bf" Jul 24 19:26:08.672979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.672965 2570 manager.go:354] Recovery completed Jul 24 19:26:08.674609 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.674577 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 19:26:08.674694 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.674644 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 19:26:08.678102 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.678081 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 19:26:08.679901 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.679881 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientMemory" Jul 24 19:26:08.679981 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.679916 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 19:26:08.679981 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.679932 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientPID" Jul 24 19:26:08.680548 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.680533 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 19:26:08.680622 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.680547 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 19:26:08.680622 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.680552 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 19:26:08.680622 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.680586 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 24 19:26:08.680873 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.680858 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cc8bf" Jul 24 19:26:08.681637 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.681622 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientMemory" Jul 24 19:26:08.681705 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.681656 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 19:26:08.681705 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.681673 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientPID" Jul 24 19:26:08.683927 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.683913 2570 policy_none.go:49] "None policy: Start" Jul 24 19:26:08.683996 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.683932 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 19:26:08.683996 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.683946 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 19:26:08.685931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.685917 2570 policy_none.go:47] "Start" Jul 24 19:26:08.716842 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.716711 2570 manager.go:337] "Starting Device Plugin manager" Jul 24 19:26:08.716936 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.716857 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 19:26:08.716936 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.716870 2570 server.go:87] "Starting device plugin registration server" Jul 24 19:26:08.717132 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.717119 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 19:26:08.717233 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.717132 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 19:26:08.717307 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.717288 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 19:26:08.717397 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.717385 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 19:26:08.717458 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.717398 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 19:26:08.717870 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.717852 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 19:26:08.717947 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.717902 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:08.818449 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.818413 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 19:26:08.819311 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.819293 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientMemory" Jul 24 19:26:08.819393 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.819324 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 19:26:08.819393 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.819334 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientPID" Jul 24 19:26:08.819393 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.819363 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-238.ec2.internal" Jul 24 19:26:08.834755 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.834713 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-238.ec2.internal" Jul 24 19:26:08.834755 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.834733 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-238.ec2.internal\": node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:08.864908 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.864889 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:08.927857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.927823 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 19:26:08.929997 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.929028 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 19:26:08.929997 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.929063 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 19:26:08.929997 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.929083 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 19:26:08.929997 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.929115 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 19:26:08.931481 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:08.931464 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 19:26:08.965342 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:08.965324 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:09.029520 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.029490 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal"] Jul 24 19:26:09.029599 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.029563 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 19:26:09.030955 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.030940 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientMemory" Jul 24 19:26:09.031058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.030970 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 19:26:09.031058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.030983 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientPID" Jul 24 19:26:09.033347 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.033334 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 19:26:09.033469 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.033457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.033505 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.033482 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 19:26:09.034036 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.034023 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientMemory" Jul 24 19:26:09.034101 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.034039 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientMemory" Jul 24 19:26:09.034101 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.034047 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 19:26:09.034101 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.034057 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientPID" Jul 24 19:26:09.034101 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.034059 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 19:26:09.034101 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.034069 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientPID" Jul 24 19:26:09.036333 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.036313 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.036333 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.036334 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 19:26:09.037004 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.036988 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientMemory" Jul 24 19:26:09.037085 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.037011 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 19:26:09.037085 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.037022 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeHasSufficientPID" Jul 24 19:26:09.050646 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.050624 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-238.ec2.internal\" not found" node="ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.053983 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.053968 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-238.ec2.internal\" not found" node="ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.066263 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.066248 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:09.066324 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.066281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6368af9d0a739b00fcf3ef01308dfffd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal\" (UID: \"6368af9d0a739b00fcf3ef01308dfffd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.066324 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.066309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6368af9d0a739b00fcf3ef01308dfffd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal\" (UID: \"6368af9d0a739b00fcf3ef01308dfffd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.066390 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.066325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a7ab8408f0b62fc7e97aef1336f6df8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-238.ec2.internal\" (UID: \"2a7ab8408f0b62fc7e97aef1336f6df8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.167335 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.167307 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:09.167446 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.167376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6368af9d0a739b00fcf3ef01308dfffd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal\" (UID: \"6368af9d0a739b00fcf3ef01308dfffd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.167446 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.167407 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6368af9d0a739b00fcf3ef01308dfffd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal\" (UID: \"6368af9d0a739b00fcf3ef01308dfffd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.167446 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.167430 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a7ab8408f0b62fc7e97aef1336f6df8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-238.ec2.internal\" (UID: \"2a7ab8408f0b62fc7e97aef1336f6df8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.167562 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.167459 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6368af9d0a739b00fcf3ef01308dfffd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal\" (UID: \"6368af9d0a739b00fcf3ef01308dfffd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.167562 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.167471 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6368af9d0a739b00fcf3ef01308dfffd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal\" (UID: \"6368af9d0a739b00fcf3ef01308dfffd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.167562 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.167498 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a7ab8408f0b62fc7e97aef1336f6df8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-238.ec2.internal\" (UID: \"2a7ab8408f0b62fc7e97aef1336f6df8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.267723 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.267690 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:09.353920 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.353889 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.357305 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.357285 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.368092 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.368070 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:09.468482 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.468410 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-238.ec2.internal\" not found" Jul 24 19:26:09.504668 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.504644 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 19:26:09.551330 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.551312 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 19:26:09.561681 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.561664 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.576140 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.576122 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 19:26:09.576938 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.576926 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" Jul 24 19:26:09.585815 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.585778 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 19:26:09.596928 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.596912 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 19:26:09.597055 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.597035 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 19:26:09.597100 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.597045 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 19:26:09.597100 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.597060 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 19:26:09.637236 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.637216 2570 apiserver.go:52] "Watching apiserver" Jul 24 19:26:09.657216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.657193 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 19:26:09.658708 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.658689 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-cx256","openshift-multus/multus-gzggx","openshift-multus/network-metrics-daemon-6l22f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66","openshift-cluster-node-tuning-operator/tuned-7zgb5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal","openshift-network-diagnostics/network-check-target-rhks5","openshift-network-operator/iptables-alerter-twhpf","openshift-ovn-kubernetes/ovnkube-node-gdlb8","kube-system/global-pull-secret-syncer-plptd","kube-system/konnectivity-agent-tw8sp","kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal","openshift-dns/node-resolver-j425q","openshift-image-registry/node-ca-xqltr"] Jul 24 19:26:09.661233 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.661215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:09.661317 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.661298 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:09.661469 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.661452 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 19:26:09.663350 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.663336 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:09.665439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.665396 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:09.665545 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.665443 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:09.665698 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.665683 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9wfvq\"" Jul 24 19:26:09.666009 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.665994 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 19:26:09.666076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.666056 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 19:26:09.667546 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.667523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:09.669538 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669523 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 19:26:09.669632 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669540 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54646590-a397-4aab-8b6f-a4de53ca338d-tmp-dir\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:09.669632 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgq85\" (UniqueName: \"kubernetes.io/projected/54646590-a397-4aab-8b6f-a4de53ca338d-kube-api-access-rgq85\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:09.669632 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:09.669632 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54646590-a397-4aab-8b6f-a4de53ca338d-hosts-file\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:09.669632 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669615 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 19:26:09.669932 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669669 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l4bwj\"" Jul 24 19:26:09.669932 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669761 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:09.669932 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.669829 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 19:26:09.671628 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.671614 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lfk6v\"" Jul 24 19:26:09.671885 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.671870 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 19:26:09.671954 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.671875 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:09.671954 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.671912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:09.673058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.673042 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 19:26:09.673941 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.673926 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 19:26:09.674067 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.674052 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9rp86\"" Jul 24 19:26:09.674143 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.674128 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 19:26:09.674247 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.674231 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:09.674320 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.674308 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 19:26:09.676297 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.676281 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 19:26:09.676558 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.676542 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:09.676634 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.676553 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 19:26:09.676634 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.676577 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4z9h8\"" Jul 24 19:26:09.676819 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.676804 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 19:26:09.677861 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.677845 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 19:26:09.677861 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.677854 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 19:26:09.678000 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.677877 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 19:26:09.678721 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.678694 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sv8pt\"" Jul 24 19:26:09.678834 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.678819 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gzggx" Jul 24 19:26:09.678951 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.678927 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 19:26:09.679080 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.679061 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 19:26:09.679219 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.679204 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 19:26:09.679388 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.679372 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 19:26:09.679472 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.679402 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 19:26:09.681123 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.681105 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:09.681289 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:09.681173 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:09.681790 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.681725 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 19:26:09.681930 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.681907 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6k9tg\"" Jul 24 19:26:09.683868 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.683849 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:09.686173 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.686144 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 19:26:09.686357 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.686344 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 19:26:09.686553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.686539 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wm7w2\"" Jul 24 19:26:09.686724 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.686710 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 19:26:09.686771 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.686753 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:09.688083 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.688046 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 19:21:10 +0000 UTC" deadline="2028-02-09 09:02:59.277621696 +0000 UTC" Jul 24 19:26:09.688083 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.688081 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13549h36m49.589543105s" Jul 24 19:26:09.688836 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.688823 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 19:26:09.689065 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.689051 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 19:26:09.689111 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.689051 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:09.689111 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.689055 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jjdz6\"" Jul 24 19:26:09.704858 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.704842 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ph7gd" Jul 24 19:26:09.713163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:09.713127 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ph7gd" Jul 24 19:26:11.409023 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.409008 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 19:26:11.415852 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.415827 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-slash\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.415929 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.415869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-env-overrides\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.415929 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.415896 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.415929 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.415916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-k8s-cni-cncf-io\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.416061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.415930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.416061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.415967 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7lzc\" (UniqueName: \"kubernetes.io/projected/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-kube-api-access-p7lzc\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.416061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.415982 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-node-log\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.416061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-socket-dir-parent\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.416061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysconfig\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgpr\" (UniqueName: \"kubernetes.io/projected/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-kube-api-access-xvgpr\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416102 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-systemd\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416131 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-cnibin\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-tuned\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416175 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416197 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysctl-conf\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-lib-modules\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416241 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-host\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.416280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4048e8da-ece3-4dcb-ae2d-f737edc52db7-agent-certs\") pod \"konnectivity-agent-tw8sp\" (UID: \"4048e8da-ece3-4dcb-ae2d-f737edc52db7\") " pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54646590-a397-4aab-8b6f-a4de53ca338d-tmp-dir\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416357 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-run-netns\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-cni-binary-copy\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416462 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-device-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416503 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-ovn\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416539 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54646590-a397-4aab-8b6f-a4de53ca338d-tmp-dir\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416542 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-log-socket\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416578 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416592 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsgj\" (UniqueName: \"kubernetes.io/projected/3e4d9366-97b2-4601-89c8-04e78cccb813-kube-api-access-zxsgj\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0350e2b-8c50-4a99-9851-4c6b60727554-host-slash\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.416627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416632 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgq85\" (UniqueName: \"kubernetes.io/projected/54646590-a397-4aab-8b6f-a4de53ca338d-kube-api-access-rgq85\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-cni-bin\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-cni-multus\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416680 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-daemon-config\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fht2\" (UniqueName: \"kubernetes.io/projected/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-kube-api-access-2fht2\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416746 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-var-lib-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416918 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-etc-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416947 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-os-release\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.416992 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-cnibin\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417009 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-var-lib-kubelet\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417033 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-kubelet\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-cni-netd\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417124 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovnkube-script-lib\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-os-release\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-multus-certs\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twdj\" (UniqueName: \"kubernetes.io/projected/c0350e2b-8c50-4a99-9851-4c6b60727554-kube-api-access-7twdj\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-modprobe-d\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7tk\" (UniqueName: \"kubernetes.io/projected/02fb5486-dd2a-48b5-898b-a60b7b88e700-kube-api-access-5r7tk\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417305 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-cni-binary-copy\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417329 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417376 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417420 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-host\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-system-cni-dir\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54646590-a397-4aab-8b6f-a4de53ca338d-hosts-file\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417540 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-systemd-units\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovnkube-config\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8132dea3-f97b-44d0-b452-bed78f859762-kubelet-config\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54646590-a397-4aab-8b6f-a4de53ca338d-hosts-file\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:11.417798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8132dea3-f97b-44d0-b452-bed78f859762-dbus\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-run\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-cni-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-netns\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0350e2b-8c50-4a99-9851-4c6b60727554-iptables-alerter-script\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417816 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9vk\" (UniqueName: \"kubernetes.io/projected/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-kube-api-access-bq9vk\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-tmp\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-hostroot\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.417963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-serviceca\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418071 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-cni-bin\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysctl-d\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-systemd\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-etc-kubernetes\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418167 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418181 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-kubelet\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418206 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-conf-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418235 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.418935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418250 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-kubernetes\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.418935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-sys\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.418935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovn-node-metrics-cert\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.418935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-system-cni-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxkwj\" (UniqueName: \"kubernetes.io/projected/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-kube-api-access-hxkwj\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.418935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.418935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.418436 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4048e8da-ece3-4dcb-ae2d-f737edc52db7-konnectivity-ca\") pod \"konnectivity-agent-tw8sp\" (UID: \"4048e8da-ece3-4dcb-ae2d-f737edc52db7\") " pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:11.422322 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.422305 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 19:26:11.422400 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.422326 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 19:26:11.422400 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.422337 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f64kz for pod openshift-network-diagnostics/network-check-target-rhks5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:11.422532 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.422407 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz podName:892c6baa-4dcb-4cda-aeb7-33dc278526a6 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:11.922384037 +0000 UTC m=+2.017873234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f64kz" (UniqueName: "kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz") pod "network-check-target-rhks5" (UID: "892c6baa-4dcb-4cda-aeb7-33dc278526a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:11.422676 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.422656 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 19:26:11.425667 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.425645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgq85\" (UniqueName: \"kubernetes.io/projected/54646590-a397-4aab-8b6f-a4de53ca338d-kube-api-access-rgq85\") pod \"node-resolver-j425q\" (UID: \"54646590-a397-4aab-8b6f-a4de53ca338d\") " pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:11.518742 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-cnibin\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.518894 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-var-lib-kubelet\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.518894 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-kubelet\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.518894 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518808 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-kubelet\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.518894 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-cnibin\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.518894 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-var-lib-kubelet\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.518894 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-cni-netd\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovnkube-script-lib\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518911 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-cni-netd\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-os-release\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-multus-certs\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twdj\" (UniqueName: \"kubernetes.io/projected/c0350e2b-8c50-4a99-9851-4c6b60727554-kube-api-access-7twdj\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.518977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-multus-certs\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-modprobe-d\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519024 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-os-release\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7tk\" (UniqueName: \"kubernetes.io/projected/02fb5486-dd2a-48b5-898b-a60b7b88e700-kube-api-access-5r7tk\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-cni-binary-copy\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519104 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-modprobe-d\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519140 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-host\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.519185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-system-cni-dir\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-systemd-units\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovnkube-config\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8132dea3-f97b-44d0-b452-bed78f859762-kubelet-config\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8132dea3-f97b-44d0-b452-bed78f859762-dbus\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-host\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-run\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519308 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-systemd-units\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-cni-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-netns\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.519363 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519383 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0350e2b-8c50-4a99-9851-4c6b60727554-iptables-alerter-script\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519408 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9vk\" (UniqueName: \"kubernetes.io/projected/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-kube-api-access-bq9vk\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.519422 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret podName:8132dea3-f97b-44d0-b452-bed78f859762 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:12.019404076 +0000 UTC m=+2.114893292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret") pod "global-pull-secret-syncer-plptd" (UID: "8132dea3-f97b-44d0-b452-bed78f859762") : object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519451 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-tmp\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519485 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-cni-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-hostroot\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.519931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovnkube-script-lib\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519526 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-serviceca\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-cni-bin\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519594 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysctl-d\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519618 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-systemd\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519642 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-etc-kubernetes\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519642 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-system-cni-dir\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519654 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-cni-binary-copy\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519707 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-kubelet\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-conf-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519769 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-hostroot\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-kubernetes\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-sys\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovn-node-metrics-cert\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-system-cni-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.520823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxkwj\" (UniqueName: \"kubernetes.io/projected/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-kube-api-access-hxkwj\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519905 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysctl-d\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4048e8da-ece3-4dcb-ae2d-f737edc52db7-konnectivity-ca\") pod \"konnectivity-agent-tw8sp\" (UID: \"4048e8da-ece3-4dcb-ae2d-f737edc52db7\") " pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519948 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-cni-bin\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519963 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-slash\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.519986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-env-overrides\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8132dea3-f97b-44d0-b452-bed78f859762-dbus\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovnkube-config\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520056 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8132dea3-f97b-44d0-b452-bed78f859762-kubelet-config\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-k8s-cni-cncf-io\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-run\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520123 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520137 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-kubernetes\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7lzc\" (UniqueName: \"kubernetes.io/projected/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-kube-api-access-p7lzc\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520173 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-node-log\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.521810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-netns\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-serviceca\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520220 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-systemd\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520228 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-run-k8s-cni-cncf-io\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520256 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-socket-dir-parent\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520280 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysconfig\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520280 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgpr\" (UniqueName: \"kubernetes.io/projected/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-kube-api-access-xvgpr\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-systemd\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520337 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-socket-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520347 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-cnibin\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-socket-dir-parent\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520376 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-node-log\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520396 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysconfig\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-systemd\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520435 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-cnibin\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e4d9366-97b2-4601-89c8-04e78cccb813-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.522734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-tuned\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520583 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-sys-fs\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520593 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-sys\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysctl-conf\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-lib-modules\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520669 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-system-cni-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-host\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4048e8da-ece3-4dcb-ae2d-f737edc52db7-agent-certs\") pod \"konnectivity-agent-tw8sp\" (UID: \"4048e8da-ece3-4dcb-ae2d-f737edc52db7\") " pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520711 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-slash\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-run-netns\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520738 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0350e2b-8c50-4a99-9851-4c6b60727554-iptables-alerter-script\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520788 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-cni-binary-copy\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520792 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-device-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520856 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.523593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-ovn\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-log-socket\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.520997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsgj\" (UniqueName: \"kubernetes.io/projected/3e4d9366-97b2-4601-89c8-04e78cccb813-kube-api-access-zxsgj\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0350e2b-8c50-4a99-9851-4c6b60727554-host-slash\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521070 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4048e8da-ece3-4dcb-ae2d-f737edc52db7-konnectivity-ca\") pod \"konnectivity-agent-tw8sp\" (UID: \"4048e8da-ece3-4dcb-ae2d-f737edc52db7\") " pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521101 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-cni-bin\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-cni-multus\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521156 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-device-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.519734 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521173 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-daemon-config\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-conf-dir\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521206 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fht2\" (UniqueName: \"kubernetes.io/projected/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-kube-api-access-2fht2\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521236 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521240 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-registration-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-etc-kubernetes\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521294 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.524378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521312 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-kubelet\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-var-lib-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521361 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-cni-bin\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521367 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-var-lib-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521401 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-etc-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521409 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-host-var-lib-cni-multus\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521118 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02fb5486-dd2a-48b5-898b-a60b7b88e700-env-overrides\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-os-release\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521549 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-os-release\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521602 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-run-ovn\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-log-socket\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e4d9366-97b2-4601-89c8-04e78cccb813-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521797 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-cni-binary-copy\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0350e2b-8c50-4a99-9851-4c6b60727554-host-slash\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.521917 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-multus-daemon-config\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.521974 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs podName:06dc9231-d8aa-465a-aa18-a1e9e46cbd18 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:12.021956666 +0000 UTC m=+2.117445864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs") pod "network-metrics-daemon-6l22f" (UID: "06dc9231-d8aa-465a-aa18-a1e9e46cbd18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-host\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.525327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522042 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522072 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-lib-modules\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522124 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-etc-openvswitch\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522160 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522164 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02fb5486-dd2a-48b5-898b-a60b7b88e700-host-run-netns\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522278 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-sysctl-conf\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.522970 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02fb5486-dd2a-48b5-898b-a60b7b88e700-ovn-node-metrics-cert\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.524222 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-etc-tuned\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.524561 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4048e8da-ece3-4dcb-ae2d-f737edc52db7-agent-certs\") pod \"konnectivity-agent-tw8sp\" (UID: \"4048e8da-ece3-4dcb-ae2d-f737edc52db7\") " pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:11.526058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.524614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-tmp\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.528832 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.528808 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7tk\" (UniqueName: \"kubernetes.io/projected/02fb5486-dd2a-48b5-898b-a60b7b88e700-kube-api-access-5r7tk\") pod \"ovnkube-node-gdlb8\" (UID: \"02fb5486-dd2a-48b5-898b-a60b7b88e700\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.528980 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.528895 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twdj\" (UniqueName: \"kubernetes.io/projected/c0350e2b-8c50-4a99-9851-4c6b60727554-kube-api-access-7twdj\") pod \"iptables-alerter-twhpf\" (UID: \"c0350e2b-8c50-4a99-9851-4c6b60727554\") " pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.529117 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.529025 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9vk\" (UniqueName: \"kubernetes.io/projected/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-kube-api-access-bq9vk\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:11.529725 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.529706 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgpr\" (UniqueName: \"kubernetes.io/projected/9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac-kube-api-access-xvgpr\") pod \"tuned-7zgb5\" (UID: \"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac\") " pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.529979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.529953 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxkwj\" (UniqueName: \"kubernetes.io/projected/9c403e81-e95d-4a0b-80f5-db75cb24f9c4-kube-api-access-hxkwj\") pod \"multus-gzggx\" (UID: \"9c403e81-e95d-4a0b-80f5-db75cb24f9c4\") " pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.530459 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.530444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fht2\" (UniqueName: \"kubernetes.io/projected/4a2b5609-a8ac-4be7-a8d6-91bce606bfd1-kube-api-access-2fht2\") pod \"node-ca-xqltr\" (UID: \"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1\") " pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.530830 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.530812 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsgj\" (UniqueName: \"kubernetes.io/projected/3e4d9366-97b2-4601-89c8-04e78cccb813-kube-api-access-zxsgj\") pod \"multus-additional-cni-plugins-cx256\" (UID: \"3e4d9366-97b2-4601-89c8-04e78cccb813\") " pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.531443 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.531420 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7lzc\" (UniqueName: \"kubernetes.io/projected/75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8-kube-api-access-p7lzc\") pod \"aws-ebs-csi-driver-node-6gs66\" (UID: \"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.613934 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.613910 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7ab8408f0b62fc7e97aef1336f6df8.slice/crio-18daa0ef50ce109722f7d2805a75ee952a3a9b62c868c79846dd59e55fda52ed WatchSource:0}: Error finding container 18daa0ef50ce109722f7d2805a75ee952a3a9b62c868c79846dd59e55fda52ed: Status 404 returned error can't find the container with id 18daa0ef50ce109722f7d2805a75ee952a3a9b62c868c79846dd59e55fda52ed Jul 24 19:26:11.614332 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.614312 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6368af9d0a739b00fcf3ef01308dfffd.slice/crio-0db151a2493d134535ac226b6a93c5ed4142bbf1508ffcf8bddd7f3d803a72ab WatchSource:0}: Error finding container 0db151a2493d134535ac226b6a93c5ed4142bbf1508ffcf8bddd7f3d803a72ab: Status 404 returned error can't find the container with id 0db151a2493d134535ac226b6a93c5ed4142bbf1508ffcf8bddd7f3d803a72ab Jul 24 19:26:11.619060 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.619047 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 19:26:11.646013 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.645954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j425q" Jul 24 19:26:11.654696 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.654677 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54646590_a397_4aab_8b6f_a4de53ca338d.slice/crio-e4c73ae43dba1e1132d23e3316bb700ddc42f1e7385a82a4077e7c57e4b0bbda WatchSource:0}: Error finding container e4c73ae43dba1e1132d23e3316bb700ddc42f1e7385a82a4077e7c57e4b0bbda: Status 404 returned error can't find the container with id e4c73ae43dba1e1132d23e3316bb700ddc42f1e7385a82a4077e7c57e4b0bbda Jul 24 19:26:11.660989 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.660975 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" Jul 24 19:26:11.668255 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.668234 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bc98e9_dc4d_4dc3_8cf9_cea727a1a8a8.slice/crio-8dcfdc4cb5b1c85deaf1a37cbdc5582e5abfff266d818a3441e7cf8637350c59 WatchSource:0}: Error finding container 8dcfdc4cb5b1c85deaf1a37cbdc5582e5abfff266d818a3441e7cf8637350c59: Status 404 returned error can't find the container with id 8dcfdc4cb5b1c85deaf1a37cbdc5582e5abfff266d818a3441e7cf8637350c59 Jul 24 19:26:11.676141 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.676127 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" Jul 24 19:26:11.683842 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.683824 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6cb2b3_2f21_4b33_8166_c3eb9fcd98ac.slice/crio-fab3d8e5738246a948c6f6f0ff5f8c150f7b512eb8ffdb403e930d2a473dc771 WatchSource:0}: Error finding container fab3d8e5738246a948c6f6f0ff5f8c150f7b512eb8ffdb403e930d2a473dc771: Status 404 returned error can't find the container with id fab3d8e5738246a948c6f6f0ff5f8c150f7b512eb8ffdb403e930d2a473dc771 Jul 24 19:26:11.687079 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.687007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqltr" Jul 24 19:26:11.693991 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.693969 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2b5609_a8ac_4be7_a8d6_91bce606bfd1.slice/crio-e0422d6e5460e1976b78ee169f14987b2be47e7b304bfe51a7602211d432ac53 WatchSource:0}: Error finding container e0422d6e5460e1976b78ee169f14987b2be47e7b304bfe51a7602211d432ac53: Status 404 returned error can't find the container with id e0422d6e5460e1976b78ee169f14987b2be47e7b304bfe51a7602211d432ac53 Jul 24 19:26:11.700053 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.700038 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:11.704762 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.704745 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cx256" Jul 24 19:26:11.707339 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.707317 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fb5486_dd2a_48b5_898b_a60b7b88e700.slice/crio-f14326d7cd81ea8e6e6e09ff18a9b536e700990fdc2f68c5af7b1c2bae53cb47 WatchSource:0}: Error finding container f14326d7cd81ea8e6e6e09ff18a9b536e700990fdc2f68c5af7b1c2bae53cb47: Status 404 returned error can't find the container with id f14326d7cd81ea8e6e6e09ff18a9b536e700990fdc2f68c5af7b1c2bae53cb47 Jul 24 19:26:11.712177 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.712157 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4d9366_97b2_4601_89c8_04e78cccb813.slice/crio-b2f83305f6ed695e951e15e668f53d583d71d5575b402c81a68acc0c4abc7a9d WatchSource:0}: Error finding container b2f83305f6ed695e951e15e668f53d583d71d5575b402c81a68acc0c4abc7a9d: Status 404 returned error can't find the container with id b2f83305f6ed695e951e15e668f53d583d71d5575b402c81a68acc0c4abc7a9d Jul 24 19:26:11.716608 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.716596 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gzggx" Jul 24 19:26:11.722940 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.722923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:11.724509 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.724490 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c403e81_e95d_4a0b_80f5_db75cb24f9c4.slice/crio-6cb55e2372d94cc4c5c2d901b16291bc41a6b0cf9e9408d4a7a0d3df46ddee6d WatchSource:0}: Error finding container 6cb55e2372d94cc4c5c2d901b16291bc41a6b0cf9e9408d4a7a0d3df46ddee6d: Status 404 returned error can't find the container with id 6cb55e2372d94cc4c5c2d901b16291bc41a6b0cf9e9408d4a7a0d3df46ddee6d Jul 24 19:26:11.731519 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:11.731501 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4048e8da_ece3_4dcb_ae2d_f737edc52db7.slice/crio-291d1f2c146e29637af8f3dd8c61782d341cd714742e1078832250703b09633f WatchSource:0}: Error finding container 291d1f2c146e29637af8f3dd8c61782d341cd714742e1078832250703b09633f: Status 404 returned error can't find the container with id 291d1f2c146e29637af8f3dd8c61782d341cd714742e1078832250703b09633f Jul 24 19:26:11.742747 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.742731 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-twhpf" Jul 24 19:26:11.752594 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.752577 2570 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-twhpf_openshift-network-operator(c0350e2b-8c50-4a99-9851-4c6b60727554): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 24 19:26:11.752656 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.752607 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-twhpf" podUID="c0350e2b-8c50-4a99-9851-4c6b60727554" Jul 24 19:26:11.923958 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:11.923902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:11.924048 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.924010 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 19:26:11.924048 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.924021 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 19:26:11.924048 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.924030 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f64kz for pod openshift-network-diagnostics/network-check-target-rhks5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:11.924138 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:11.924067 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz podName:892c6baa-4dcb-4cda-aeb7-33dc278526a6 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:12.924055537 +0000 UTC m=+3.019544733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f64kz" (UniqueName: "kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz") pod "network-check-target-rhks5" (UID: "892c6baa-4dcb-4cda-aeb7-33dc278526a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:12.025266 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.025229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:12.025433 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.025304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:12.025433 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.025362 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:12.025433 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.025425 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret podName:8132dea3-f97b-44d0-b452-bed78f859762 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:13.025410793 +0000 UTC m=+3.120899983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret") pod "global-pull-secret-syncer-plptd" (UID: "8132dea3-f97b-44d0-b452-bed78f859762") : object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:12.025433 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.025426 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:12.025737 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.025500 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs podName:06dc9231-d8aa-465a-aa18-a1e9e46cbd18 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:13.025480936 +0000 UTC m=+3.120970140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs") pod "network-metrics-daemon-6l22f" (UID: "06dc9231-d8aa-465a-aa18-a1e9e46cbd18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:12.361179 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.361119 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 19:21:11 +0000 UTC" deadline="2028-03-21 02:58:16.726056871 +0000 UTC" Jul 24 19:26:12.361179 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.361176 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14527h32m4.364884214s" Jul 24 19:26:12.394790 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.394766 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 19:26:12.585520 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.585450 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gzggx" event={"ID":"9c403e81-e95d-4a0b-80f5-db75cb24f9c4","Type":"ContainerStarted","Data":"6cb55e2372d94cc4c5c2d901b16291bc41a6b0cf9e9408d4a7a0d3df46ddee6d"} Jul 24 19:26:12.592370 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.592315 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" event={"ID":"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac","Type":"ContainerStarted","Data":"fab3d8e5738246a948c6f6f0ff5f8c150f7b512eb8ffdb403e930d2a473dc771"} Jul 24 19:26:12.596670 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.596644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" event={"ID":"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8","Type":"ContainerStarted","Data":"8dcfdc4cb5b1c85deaf1a37cbdc5582e5abfff266d818a3441e7cf8637350c59"} Jul 24 19:26:12.610437 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.610344 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j425q" event={"ID":"54646590-a397-4aab-8b6f-a4de53ca338d","Type":"ContainerStarted","Data":"e4c73ae43dba1e1132d23e3316bb700ddc42f1e7385a82a4077e7c57e4b0bbda"} Jul 24 19:26:12.616446 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.616335 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" event={"ID":"6368af9d0a739b00fcf3ef01308dfffd","Type":"ContainerStarted","Data":"0db151a2493d134535ac226b6a93c5ed4142bbf1508ffcf8bddd7f3d803a72ab"} Jul 24 19:26:12.646923 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.646894 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-twhpf" event={"ID":"c0350e2b-8c50-4a99-9851-4c6b60727554","Type":"ContainerStarted","Data":"f05f8008405d956d2bf9c813c882c0685228c8b11a153e29be21d40e4a2a45e9"} Jul 24 19:26:12.656394 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.656366 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-twhpf" podUID="c0350e2b-8c50-4a99-9851-4c6b60727554" Jul 24 19:26:12.665028 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.664965 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tw8sp" event={"ID":"4048e8da-ece3-4dcb-ae2d-f737edc52db7","Type":"ContainerStarted","Data":"291d1f2c146e29637af8f3dd8c61782d341cd714742e1078832250703b09633f"} Jul 24 19:26:12.668933 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.668877 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerStarted","Data":"b2f83305f6ed695e951e15e668f53d583d71d5575b402c81a68acc0c4abc7a9d"} Jul 24 19:26:12.690553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.690422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"f14326d7cd81ea8e6e6e09ff18a9b536e700990fdc2f68c5af7b1c2bae53cb47"} Jul 24 19:26:12.709993 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.709966 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqltr" event={"ID":"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1","Type":"ContainerStarted","Data":"e0422d6e5460e1976b78ee169f14987b2be47e7b304bfe51a7602211d432ac53"} Jul 24 19:26:12.724676 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.724648 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" event={"ID":"2a7ab8408f0b62fc7e97aef1336f6df8","Type":"ContainerStarted","Data":"18daa0ef50ce109722f7d2805a75ee952a3a9b62c868c79846dd59e55fda52ed"} Jul 24 19:26:12.933277 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:12.932624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:12.933277 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.932812 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 19:26:12.933277 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.932829 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 19:26:12.933277 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.932841 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f64kz for pod openshift-network-diagnostics/network-check-target-rhks5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:12.933277 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:12.932890 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz podName:892c6baa-4dcb-4cda-aeb7-33dc278526a6 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:14.932874564 +0000 UTC m=+5.028363759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f64kz" (UniqueName: "kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz") pod "network-check-target-rhks5" (UID: "892c6baa-4dcb-4cda-aeb7-33dc278526a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:13.032994 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.032956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:13.033236 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.033063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:13.033236 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:13.033199 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:13.033353 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:13.033257 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret podName:8132dea3-f97b-44d0-b452-bed78f859762 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:15.033239239 +0000 UTC m=+5.128728426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret") pod "global-pull-secret-syncer-plptd" (UID: "8132dea3-f97b-44d0-b452-bed78f859762") : object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:13.033696 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:13.033676 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:13.033796 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:13.033733 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs podName:06dc9231-d8aa-465a-aa18-a1e9e46cbd18 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:15.0337164 +0000 UTC m=+5.129205593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs") pod "network-metrics-daemon-6l22f" (UID: "06dc9231-d8aa-465a-aa18-a1e9e46cbd18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:13.146590 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.146561 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 19:26:13.361915 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.361872 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 19:21:11 +0000 UTC" deadline="2028-01-04 10:28:13.237429732 +0000 UTC" Jul 24 19:26:13.361915 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.361912 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12687h1m59.875522074s" Jul 24 19:26:13.576249 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.575390 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:13.576249 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:13.575535 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:13.576249 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.575906 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:13.576249 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:13.575988 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:13.576249 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:13.576048 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:13.576249 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:13.576123 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:14.042209 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:14.042163 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 19:26:14.952254 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:14.952217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:14.952785 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:14.952434 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 19:26:14.952785 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:14.952456 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 19:26:14.952785 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:14.952482 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f64kz for pod openshift-network-diagnostics/network-check-target-rhks5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:14.952785 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:14.952547 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz podName:892c6baa-4dcb-4cda-aeb7-33dc278526a6 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:18.952529314 +0000 UTC m=+9.048018506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f64kz" (UniqueName: "kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz") pod "network-check-target-rhks5" (UID: "892c6baa-4dcb-4cda-aeb7-33dc278526a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:15.053675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:15.053638 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:15.053829 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:15.053695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:15.053829 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:15.053791 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:15.053919 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:15.053851 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs podName:06dc9231-d8aa-465a-aa18-a1e9e46cbd18 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:19.053834071 +0000 UTC m=+9.149323260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs") pod "network-metrics-daemon-6l22f" (UID: "06dc9231-d8aa-465a-aa18-a1e9e46cbd18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:15.053919 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:15.053790 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:15.053919 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:15.053901 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret podName:8132dea3-f97b-44d0-b452-bed78f859762 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:19.053889183 +0000 UTC m=+9.149378381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret") pod "global-pull-secret-syncer-plptd" (UID: "8132dea3-f97b-44d0-b452-bed78f859762") : object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:15.575335 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:15.575302 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:15.575335 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:15.575332 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:15.575582 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:15.575344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:15.575582 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:15.575450 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:15.575690 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:15.575582 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:15.575690 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:15.575670 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:17.575051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:17.575018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:17.575531 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:17.575177 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:17.575531 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:17.575179 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:17.575531 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:17.575259 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:17.575531 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:17.575291 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:17.575531 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:17.575347 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:18.987152 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:18.987102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:18.987633 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:18.987298 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 19:26:18.987633 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:18.987320 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 19:26:18.987633 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:18.987332 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f64kz for pod openshift-network-diagnostics/network-check-target-rhks5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:18.987633 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:18.987386 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz podName:892c6baa-4dcb-4cda-aeb7-33dc278526a6 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:26.987367588 +0000 UTC m=+17.082856794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f64kz" (UniqueName: "kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz") pod "network-check-target-rhks5" (UID: "892c6baa-4dcb-4cda-aeb7-33dc278526a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:19.088049 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:19.088011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:19.088197 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:19.088079 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:19.088197 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:19.088141 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:19.088317 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:19.088211 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret podName:8132dea3-f97b-44d0-b452-bed78f859762 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:27.088191354 +0000 UTC m=+17.183680538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret") pod "global-pull-secret-syncer-plptd" (UID: "8132dea3-f97b-44d0-b452-bed78f859762") : object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:19.088317 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:19.088215 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:19.088317 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:19.088271 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs podName:06dc9231-d8aa-465a-aa18-a1e9e46cbd18 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:27.088254623 +0000 UTC m=+17.183743812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs") pod "network-metrics-daemon-6l22f" (UID: "06dc9231-d8aa-465a-aa18-a1e9e46cbd18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:19.575787 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:19.575303 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:19.575787 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:19.575439 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:19.575787 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:19.575476 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:19.575787 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:19.575610 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:19.575787 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:19.575672 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:19.575787 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:19.575747 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:21.575350 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:21.575313 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:21.575801 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:21.575425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:21.575801 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:21.575544 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:21.575801 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:21.575431 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:21.575801 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:21.575431 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:21.575801 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:21.575653 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:23.575418 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:23.575384 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:23.575857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:23.575425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:23.575857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:23.575422 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:23.575857 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:23.575536 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:23.575857 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:23.575583 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:23.575857 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:23.575635 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:25.575552 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:25.575506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:25.575552 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:25.575536 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:25.576038 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:25.575523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:25.576038 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:25.575633 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:25.576038 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:25.575679 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:25.576038 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:25.575787 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:27.048949 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:27.048912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:27.049400 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.049079 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 19:26:27.049400 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.049099 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 19:26:27.049400 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.049116 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f64kz for pod openshift-network-diagnostics/network-check-target-rhks5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:27.049400 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.049184 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz podName:892c6baa-4dcb-4cda-aeb7-33dc278526a6 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:43.049169362 +0000 UTC m=+33.144658550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f64kz" (UniqueName: "kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz") pod "network-check-target-rhks5" (UID: "892c6baa-4dcb-4cda-aeb7-33dc278526a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:27.149710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:27.149677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:27.149710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:27.149718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:27.149951 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.149831 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:27.149951 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.149889 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs podName:06dc9231-d8aa-465a-aa18-a1e9e46cbd18 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:43.149875603 +0000 UTC m=+33.245364787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs") pod "network-metrics-daemon-6l22f" (UID: "06dc9231-d8aa-465a-aa18-a1e9e46cbd18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:27.149951 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.149831 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:27.150102 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.149998 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret podName:8132dea3-f97b-44d0-b452-bed78f859762 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:43.149976255 +0000 UTC m=+33.245465454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret") pod "global-pull-secret-syncer-plptd" (UID: "8132dea3-f97b-44d0-b452-bed78f859762") : object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:27.574973 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:27.574933 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:27.575164 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:27.575074 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:27.575164 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:27.575122 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:27.575294 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.575078 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:27.575294 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.575209 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:27.575382 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:27.575298 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:29.575035 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:29.574914 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:29.575035 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:29.574953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:29.575035 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:29.574959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:29.575551 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:29.575036 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:29.575551 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:29.575077 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:29.575551 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:29.575187 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:31.574722 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.574697 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:31.575391 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:31.574812 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:31.575391 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.574876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:31.575391 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:31.574933 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:31.575391 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.574979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:31.575391 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:31.575039 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:31.764923 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.764704 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tw8sp" event={"ID":"4048e8da-ece3-4dcb-ae2d-f737edc52db7","Type":"ContainerStarted","Data":"37ed0adcd892bfb3f1fbf7991394ee0105b2762a1c3784526f546e9a1f540459"} Jul 24 19:26:31.766440 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.766410 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e4d9366-97b2-4601-89c8-04e78cccb813" containerID="cf4b0c9595b13e4cfa06e011307cd5260010b5b894624efe8bc89653c2cc708d" exitCode=0 Jul 24 19:26:31.766547 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.766487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerDied","Data":"cf4b0c9595b13e4cfa06e011307cd5260010b5b894624efe8bc89653c2cc708d"} Jul 24 19:26:31.768827 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.768804 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"5c8906ebd34c75a3bf5687f57e1cded5c5fa9e547fb1953e8b9e578badcf97f5"} Jul 24 19:26:31.768904 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.768877 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"aaef1280035a13c45accc36f511f7ac3be27663889eef8b2a66aa9074ce9b215"} Jul 24 19:26:31.770403 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.770356 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqltr" event={"ID":"4a2b5609-a8ac-4be7-a8d6-91bce606bfd1","Type":"ContainerStarted","Data":"d0c1bace48d765a3f564f0ba502d623ff81649e294c5af0be1efb23f59b866a2"} Jul 24 19:26:31.771814 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.771787 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" event={"ID":"2a7ab8408f0b62fc7e97aef1336f6df8","Type":"ContainerStarted","Data":"c6b7943bd45c228d0bf0b6c3323a945586dc1cc483fcc237776117ff194870da"} Jul 24 19:26:31.773391 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.773366 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gzggx" event={"ID":"9c403e81-e95d-4a0b-80f5-db75cb24f9c4","Type":"ContainerStarted","Data":"56c2ad6c5cd73be6b139638b0cbc3bb7df9a82d6f00b3bda5dc56f03af9463ee"} Jul 24 19:26:31.774859 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.774841 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" event={"ID":"9e6cb2b3-2f21-4b33-8166-c3eb9fcd98ac","Type":"ContainerStarted","Data":"871219c561c1ad2f9e6359ad6ecc36a4a803ddd847690471ce192375f51d7db1"} Jul 24 19:26:31.776202 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.776176 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" event={"ID":"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8","Type":"ContainerStarted","Data":"c7739cc3969bb7ee2d52dcc0c55e020b86d62001812e7dad8ca64fe10af79dd5"} Jul 24 19:26:31.777523 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.777500 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j425q" event={"ID":"54646590-a397-4aab-8b6f-a4de53ca338d","Type":"ContainerStarted","Data":"02f1f0d0f2ae8ad9250c1da6cf9d9bf8fb77bff826e6f68be431cf2622d6e006"} Jul 24 19:26:31.778803 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.778784 2570 generic.go:356] "Generic (PLEG): container finished" podID="6368af9d0a739b00fcf3ef01308dfffd" containerID="e431e399c0c5e1dd5039a1397ec2c3005589b84649fbfffefba34ba55735ec88" exitCode=0 Jul 24 19:26:31.778887 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.778810 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" event={"ID":"6368af9d0a739b00fcf3ef01308dfffd","Type":"ContainerDied","Data":"e431e399c0c5e1dd5039a1397ec2c3005589b84649fbfffefba34ba55735ec88"} Jul 24 19:26:31.783587 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.783552 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tw8sp" podStartSLOduration=2.788996738 podStartE2EDuration="21.783540747s" totalImagesPullingTime="18.994544009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:31.783309802 +0000 UTC m=+21.878799007" watchObservedRunningTime="2026-07-24 19:26:31.783540747 +0000 UTC m=+21.879029953" Jul 24 19:26:31.798758 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.798721 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-238.ec2.internal" podStartSLOduration=20.798708383 podStartE2EDuration="20.798708383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:31.798312183 +0000 UTC m=+21.893801388" watchObservedRunningTime="2026-07-24 19:26:31.798708383 +0000 UTC m=+21.894197587" Jul 24 19:26:31.813867 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.813828 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j425q" podStartSLOduration=2.742447003 podStartE2EDuration="21.813812334s" totalImagesPullingTime="19.071365331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:31.813274875 +0000 UTC m=+21.908764093" watchObservedRunningTime="2026-07-24 19:26:31.813812334 +0000 UTC m=+21.909301541" Jul 24 19:26:31.861839 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.861800 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gzggx" podStartSLOduration=2.834329078 podStartE2EDuration="21.861785696s" totalImagesPullingTime="19.027456618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:31.861366406 +0000 UTC m=+21.956855612" watchObservedRunningTime="2026-07-24 19:26:31.861785696 +0000 UTC m=+21.957274905" Jul 24 19:26:31.877607 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.877566 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7zgb5" podStartSLOduration=2.835665907 podStartE2EDuration="21.877553825s" totalImagesPullingTime="19.041887918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:31.877340066 +0000 UTC m=+21.972829273" watchObservedRunningTime="2026-07-24 19:26:31.877553825 +0000 UTC m=+21.973043033" Jul 24 19:26:31.891576 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:31.891542 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xqltr" podStartSLOduration=2.859085468 podStartE2EDuration="21.891530379s" totalImagesPullingTime="19.032444911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:31.891496054 +0000 UTC m=+21.986985259" watchObservedRunningTime="2026-07-24 19:26:31.891530379 +0000 UTC m=+21.987019583" Jul 24 19:26:32.064016 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.063990 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:32.064439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.064423 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:32.782731 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.782688 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-twhpf" event={"ID":"c0350e2b-8c50-4a99-9851-4c6b60727554","Type":"ContainerStarted","Data":"ab2790277bce52b60f4f32e84b21d766993f0ded64e46710989ba077553e65ef"} Jul 24 19:26:32.785812 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.785783 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"667cbb944825f0b3e8cfb7f56a53dadc351365096f023b31f4107d828d82893d"} Jul 24 19:26:32.785939 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.785864 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"65c82e780179833295dbf9e4c39fce1c527a2b444937ceabd59081ef1fc6708d"} Jul 24 19:26:32.785939 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.785913 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"ad6addc096eeadbbf9c811efec2f0c23461f95566514c5c1058930daaefad42b"} Jul 24 19:26:32.786049 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.785960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"c43b5e21552be1ed4daff1d6a2a7f1a8851535e7a9497081401eb58ba8b6a04b"} Jul 24 19:26:32.798846 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.798803 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-twhpf" podStartSLOduration=22.798788377 podStartE2EDuration="22.798788377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-24 19:26:32.798326054 +0000 UTC m=+22.893815260" watchObservedRunningTime="2026-07-24 19:26:32.798788377 +0000 UTC m=+22.894277583" Jul 24 19:26:32.933745 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:32.933610 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 19:26:33.374772 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.374603 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T19:26:32.933628426Z","UUID":"3332a235-b455-4bbe-8aaa-830964ce2099","Handler":null,"Name":"","Endpoint":""} Jul 24 19:26:33.376112 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.376081 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 19:26:33.376249 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.376124 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 19:26:33.575246 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.575212 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:33.575246 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.575243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:33.575617 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.575252 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:33.575617 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:33.575338 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:33.575617 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:33.575418 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:33.575617 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:33.575519 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:33.790287 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.790224 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" event={"ID":"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8","Type":"ContainerStarted","Data":"968b158b195b75a8c31a0840b895cfa452f51ef9620417afb5466cbc2cdcf376"} Jul 24 19:26:33.792382 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.792357 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" event={"ID":"6368af9d0a739b00fcf3ef01308dfffd","Type":"ContainerStarted","Data":"c3d871660bc8da32b064d1b99e3f9307b0051f27d604051e1ecaf76c19f9e588"} Jul 24 19:26:33.792382 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.792373 2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 19:26:33.811675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:33.811537 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-238.ec2.internal" podStartSLOduration=22.811520255 podStartE2EDuration="22.811520255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:33.810572174 +0000 UTC m=+23.906061379" watchObservedRunningTime="2026-07-24 19:26:33.811520255 +0000 UTC m=+23.907009465" Jul 24 19:26:34.798332 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:34.798294 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"8a57238942b6e07257874b3d44019e351ba164b5dd7a398f5951834c5bb65dcd"} Jul 24 19:26:34.800563 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:34.800536 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" event={"ID":"75bc98e9-dc4d-4dc3-8cf9-cea727a1a8a8","Type":"ContainerStarted","Data":"1e81bd6674dc8cef55695d3036407ab03d57d715c53a4ed83f14988535350e12"} Jul 24 19:26:34.822310 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:34.822270 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gs66" podStartSLOduration=3.304048939 podStartE2EDuration="24.82225658s" totalImagesPullingTime="21.518207641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:34.82118729 +0000 UTC m=+24.916676497" watchObservedRunningTime="2026-07-24 19:26:34.82225658 +0000 UTC m=+24.917745787" Jul 24 19:26:35.575407 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:35.575367 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:35.575407 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:35.575401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:35.575664 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:35.575405 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:35.575664 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:35.575510 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:35.575664 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:35.575636 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:35.575803 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:35.575739 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:36.806320 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.806290 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e4d9366-97b2-4601-89c8-04e78cccb813" containerID="96e3a9e76902870638266cd592dfa5aa83493531647bbdcb2c42e00c25cbc41d" exitCode=0 Jul 24 19:26:36.807082 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.806387 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerDied","Data":"96e3a9e76902870638266cd592dfa5aa83493531647bbdcb2c42e00c25cbc41d"} Jul 24 19:26:36.809997 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.809976 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" event={"ID":"02fb5486-dd2a-48b5-898b-a60b7b88e700","Type":"ContainerStarted","Data":"144e0d5b80f971d4a9d8c07599fc6f6855f12a848b5bc48fccccd6554b798ebd"} Jul 24 19:26:36.810285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.810269 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:36.810484 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.810333 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:36.810484 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.810382 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:36.825484 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.825447 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:36.825587 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:36.825575 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:26:37.575235 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:37.575069 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:37.575383 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:37.575074 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:37.575383 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:37.575295 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:37.575505 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:37.575393 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:37.575505 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:37.575070 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:37.575505 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:37.575492 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:38.397582 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.397526 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" podStartSLOduration=8.778136516 podStartE2EDuration="28.397510622s" totalImagesPullingTime="19.619374106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:36.901367769 +0000 UTC m=+26.996856976" watchObservedRunningTime="2026-07-24 19:26:38.397510622 +0000 UTC m=+28.492999829" Jul 24 19:26:38.397979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.397738 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plptd"] Jul 24 19:26:38.397979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.397815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:38.397979 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:38.397894 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:38.401533 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.401507 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6l22f"] Jul 24 19:26:38.401677 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.401605 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:38.401742 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:38.401706 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:38.402328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.402301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rhks5"] Jul 24 19:26:38.402433 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.402387 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:38.402513 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:38.402487 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:38.815796 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.815765 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e4d9366-97b2-4601-89c8-04e78cccb813" containerID="3efcc275177edcf647fc7c229d423a03ddb8860d40b9cfcc36ff20b1703237d7" exitCode=0 Jul 24 19:26:38.815954 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:38.815852 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerDied","Data":"3efcc275177edcf647fc7c229d423a03ddb8860d40b9cfcc36ff20b1703237d7"} Jul 24 19:26:39.574593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:39.574555 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:39.575115 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:39.574650 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:40.202542 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:40.202512 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:40.202690 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:40.202677 2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 19:26:40.203288 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:40.203266 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tw8sp" Jul 24 19:26:40.575792 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:40.575765 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:40.576133 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:40.575851 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:40.576133 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:40.575954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:40.576133 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:40.576055 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:40.822923 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:40.822893 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e4d9366-97b2-4601-89c8-04e78cccb813" containerID="a0462b85ce22d0aac0dcc2271c38b371ee44ac3a1250c8475af2430e6f8b593d" exitCode=0 Jul 24 19:26:40.823094 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:40.822973 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerDied","Data":"a0462b85ce22d0aac0dcc2271c38b371ee44ac3a1250c8475af2430e6f8b593d"} Jul 24 19:26:41.575598 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:41.575568 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:41.575752 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:41.575688 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:42.574942 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:42.574913 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:42.575362 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:42.575013 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:42.575362 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:42.575067 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:42.575362 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:42.575202 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:43.070445 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:43.070414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:43.070616 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.070538 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 19:26:43.070616 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.070556 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 19:26:43.070616 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.070566 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f64kz for pod openshift-network-diagnostics/network-check-target-rhks5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:43.070616 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.070611 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz podName:892c6baa-4dcb-4cda-aeb7-33dc278526a6 nodeName:}" failed. No retries permitted until 2026-07-24 19:27:15.070598603 +0000 UTC m=+65.166087803 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f64kz" (UniqueName: "kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz") pod "network-check-target-rhks5" (UID: "892c6baa-4dcb-4cda-aeb7-33dc278526a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 19:26:43.171290 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:43.171219 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:43.171290 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:43.171253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:43.171529 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.171377 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:43.171529 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.171418 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:43.171529 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.171441 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret podName:8132dea3-f97b-44d0-b452-bed78f859762 nodeName:}" failed. No retries permitted until 2026-07-24 19:27:15.17142397 +0000 UTC m=+65.266913166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret") pod "global-pull-secret-syncer-plptd" (UID: "8132dea3-f97b-44d0-b452-bed78f859762") : object "kube-system"/"original-pull-secret" not registered Jul 24 19:26:43.171529 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.171480 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs podName:06dc9231-d8aa-465a-aa18-a1e9e46cbd18 nodeName:}" failed. No retries permitted until 2026-07-24 19:27:15.171453524 +0000 UTC m=+65.266942708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs") pod "network-metrics-daemon-6l22f" (UID: "06dc9231-d8aa-465a-aa18-a1e9e46cbd18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 19:26:43.574622 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:43.574592 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:43.574792 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:43.574697 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plptd" podUID="8132dea3-f97b-44d0-b452-bed78f859762" Jul 24 19:26:44.575442 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.575259 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:44.575973 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.575314 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:44.575973 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:44.575563 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l22f" podUID="06dc9231-d8aa-465a-aa18-a1e9e46cbd18" Jul 24 19:26:44.575973 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:44.575707 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhks5" podUID="892c6baa-4dcb-4cda-aeb7-33dc278526a6" Jul 24 19:26:44.756206 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.756176 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-238.ec2.internal" event="NodeReady" Jul 24 19:26:44.756364 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.756332 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 19:26:44.869048 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.868955 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f"] Jul 24 19:26:44.898877 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.898802 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2"] Jul 24 19:26:44.898877 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.898878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:44.904653 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.904252 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 19:26:44.904653 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.904361 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 19:26:44.904653 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.904507 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 19:26:44.904915 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.904894 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hgmb7\"" Jul 24 19:26:44.906845 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.906676 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 19:26:44.914774 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.914754 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-s6q2j"] Jul 24 19:26:44.914888 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.914871 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" Jul 24 19:26:44.926818 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.926802 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 19:26:44.926919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.926813 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b8z4n\"" Jul 24 19:26:44.931441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.931411 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:44.931836 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.931813 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2"] Jul 24 19:26:44.932019 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.932000 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:44.936394 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.936374 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 19:26:44.936779 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.936586 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 19:26:44.936779 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.936629 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 19:26:44.937040 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.937018 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-v229b\"" Jul 24 19:26:44.939365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.939345 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 19:26:44.944526 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.944505 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 19:26:44.949157 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.949134 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-frwvb"] Jul 24 19:26:44.949494 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.949365 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:44.953301 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.953282 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 19:26:44.953439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.953421 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 19:26:44.953527 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.953428 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7v49n\"" Jul 24 19:26:44.953734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.953717 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 19:26:44.953734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.953730 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:44.975345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.975322 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56947db777-2bfh7"] Jul 24 19:26:44.975506 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.975488 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:44.987715 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.986827 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:44.987715 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.986905 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 19:26:44.987715 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.987251 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-89sg4\"" Jul 24 19:26:44.987715 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.987363 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 19:26:44.987715 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.987569 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 19:26:44.988128 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.988104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:44.988199 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.988174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7267d569-0982-4797-af25-32d1752be10b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:44.988245 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.988203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfb66\" (UniqueName: \"kubernetes.io/projected/7267d569-0982-4797-af25-32d1752be10b-kube-api-access-gfb66\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:44.997315 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.997287 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f69ccb89-9t2qh"] Jul 24 19:26:44.997456 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.997438 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:44.999629 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:44.999610 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 19:26:45.000350 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.000336 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n8wrr\"" Jul 24 19:26:45.001458 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.001438 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 19:26:45.001576 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.001509 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 19:26:45.002759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.002743 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 19:26:45.004223 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.004191 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 19:26:45.004379 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.004365 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 19:26:45.004677 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.004662 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 19:26:45.015552 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.015515 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2"] Jul 24 19:26:45.015552 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.015551 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2"] Jul 24 19:26:45.015726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.015571 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f"] Jul 24 19:26:45.015726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.015589 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f69ccb89-9t2qh"] Jul 24 19:26:45.015726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.015610 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56947db777-2bfh7"] Jul 24 19:26:45.015726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.015637 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mnvj7"] Jul 24 19:26:45.015726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.015673 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.020563 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.020529 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 19:26:45.021585 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.021568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 19:26:45.023637 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.023616 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jcdcc\"" Jul 24 19:26:45.025855 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.025835 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 19:26:45.028338 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.028304 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 19:26:45.043633 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.043596 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx"] Jul 24 19:26:45.043994 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.043957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.050061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.050035 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 19:26:45.050520 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.050498 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 19:26:45.050682 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.050604 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 19:26:45.050891 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.050876 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8tvpg\"" Jul 24 19:26:45.057979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.057956 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ghxhc"] Jul 24 19:26:45.058177 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.058126 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:45.065168 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.065151 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:45.065557 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.065342 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 19:26:45.065557 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.065393 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d4scs\"" Jul 24 19:26:45.065557 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.065459 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 19:26:45.074445 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.074426 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst"] Jul 24 19:26:45.074586 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.074571 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.082610 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.082595 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 19:26:45.082857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.082837 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75276\"" Jul 24 19:26:45.083143 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.083124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 19:26:45.089133 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/faeb92d0-d3b8-4602-95ce-941edc7337dd-serving-cert\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.089216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089141 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.089216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-stats-auth\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.089216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vkx\" (UniqueName: \"kubernetes.io/projected/9996735c-0afe-414c-b463-b2f4c223d41c-kube-api-access-b9vkx\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.089336 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.089336 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faeb92d0-d3b8-4602-95ce-941edc7337dd-tmp\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.089336 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/faeb92d0-d3b8-4602-95ce-941edc7337dd-snapshots\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.089336 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089329 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de633ac5-2c17-401d-8529-c02d480d9f25-serving-cert\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.089560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkpr\" (UniqueName: \"kubernetes.io/projected/d1cdec05-18f6-4d64-ac29-65b77e5ff36b-kube-api-access-hfkpr\") pod \"volume-data-source-validator-549c6d54cf-lgkl2\" (UID: \"d1cdec05-18f6-4d64-ac29-65b77e5ff36b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" Jul 24 19:26:45.089560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6mf\" (UniqueName: \"kubernetes.io/projected/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-kube-api-access-5v6mf\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.089560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089486 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faeb92d0-d3b8-4602-95ce-941edc7337dd-service-ca-bundle\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.089560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tbk\" (UniqueName: \"kubernetes.io/projected/de633ac5-2c17-401d-8529-c02d480d9f25-kube-api-access-96tbk\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.089758 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089569 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2b4l\" (UniqueName: \"kubernetes.io/projected/faeb92d0-d3b8-4602-95ce-941edc7337dd-kube-api-access-t2b4l\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.089758 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089630 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faeb92d0-d3b8-4602-95ce-941edc7337dd-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.089758 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:45.089758 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089704 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7267d569-0982-4797-af25-32d1752be10b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:45.089758 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089729 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfb66\" (UniqueName: \"kubernetes.io/projected/7267d569-0982-4797-af25-32d1752be10b-kube-api-access-gfb66\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:45.090000 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.089792 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:45.090000 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089825 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-default-certificate\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.090000 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.089838 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls podName:7267d569-0982-4797-af25-32d1752be10b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:45.589825426 +0000 UTC m=+35.685314611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5xb5f" (UID: "7267d569-0982-4797-af25-32d1752be10b") : secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:45.090000 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de633ac5-2c17-401d-8529-c02d480d9f25-config\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.090000 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.090000 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.090000 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.089961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de633ac5-2c17-401d-8529-c02d480d9f25-trusted-ca\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.090366 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.090349 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7267d569-0982-4797-af25-32d1752be10b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:45.097881 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.097846 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-frwvb"] Jul 24 19:26:45.097972 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.097895 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-s6q2j"] Jul 24 19:26:45.097972 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.097920 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp"] Jul 24 19:26:45.097972 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.097957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.101674 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.101654 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 19:26:45.101769 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.101693 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 19:26:45.101939 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.101916 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:45.102532 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.102417 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 19:26:45.102532 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.102425 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5kklg\"" Jul 24 19:26:45.110893 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.110873 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfb66\" (UniqueName: \"kubernetes.io/projected/7267d569-0982-4797-af25-32d1752be10b-kube-api-access-gfb66\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:45.122103 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.122053 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnvj7"] Jul 24 19:26:45.122103 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.122081 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ghxhc"] Jul 24 19:26:45.122103 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.122103 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst"] Jul 24 19:26:45.122309 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.122124 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx"] Jul 24 19:26:45.122309 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.122153 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp"] Jul 24 19:26:45.122309 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.122187 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" Jul 24 19:26:45.125346 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.125330 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ssvrx\"" Jul 24 19:26:45.125804 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.125786 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 19:26:45.125804 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.125795 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 19:26:45.190560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/938f432d-fa76-41c9-b82b-25f85fbb30be-tmp-dir\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.190560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-kube-api-access-ltf6j\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.190799 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190593 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6mf\" (UniqueName: \"kubernetes.io/projected/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-kube-api-access-5v6mf\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.190799 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9q79\" (UniqueName: \"kubernetes.io/projected/028f41d9-d9c3-44ed-85af-4c7078cf2cd4-kube-api-access-h9q79\") pod \"ingress-canary-mnvj7\" (UID: \"028f41d9-d9c3-44ed-85af-4c7078cf2cd4\") " pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.190799 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgplx\" (UniqueName: \"kubernetes.io/projected/7da57dfc-bfd9-4fa4-a39f-a90957d47706-kube-api-access-tgplx\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:45.190946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190798 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tbk\" (UniqueName: \"kubernetes.io/projected/de633ac5-2c17-401d-8529-c02d480d9f25-kube-api-access-96tbk\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.190946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190828 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/028f41d9-d9c3-44ed-85af-4c7078cf2cd4-cert\") pod \"ingress-canary-mnvj7\" (UID: \"028f41d9-d9c3-44ed-85af-4c7078cf2cd4\") " pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.190946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.190946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190862 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-ca-trust-extracted\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.190946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-certificates\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.190946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190939 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-bound-sa-token\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.191234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.190981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2b4l\" (UniqueName: \"kubernetes.io/projected/faeb92d0-d3b8-4602-95ce-941edc7337dd-kube-api-access-t2b4l\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.191234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faeb92d0-d3b8-4602-95ce-941edc7337dd-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.191234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191063 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-serving-cert\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.191234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.191234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/faeb92d0-d3b8-4602-95ce-941edc7337dd-serving-cert\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.191234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.191293 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:45.691268737 +0000 UTC m=+35.786757939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : configmap references non-existent config key: service-ca.crt Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vkx\" (UniqueName: \"kubernetes.io/projected/9996735c-0afe-414c-b463-b2f4c223d41c-kube-api-access-b9vkx\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-trusted-ca\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faeb92d0-d3b8-4602-95ce-941edc7337dd-tmp\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/faeb92d0-d3b8-4602-95ce-941edc7337dd-snapshots\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191519 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de633ac5-2c17-401d-8529-c02d480d9f25-serving-cert\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.191540 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.191525 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkpr\" (UniqueName: \"kubernetes.io/projected/d1cdec05-18f6-4d64-ac29-65b77e5ff36b-kube-api-access-hfkpr\") pod \"volume-data-source-validator-549c6d54cf-lgkl2\" (UID: \"d1cdec05-18f6-4d64-ac29-65b77e5ff36b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.191574 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:45.691558494 +0000 UTC m=+35.787047679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : secret "router-metrics-certs-default" not found Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/938f432d-fa76-41c9-b82b-25f85fbb30be-config-volume\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faeb92d0-d3b8-4602-95ce-941edc7337dd-service-ca-bundle\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-image-registry-private-configuration\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191707 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blsx4\" (UniqueName: \"kubernetes.io/projected/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-kube-api-access-blsx4\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191739 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de633ac5-2c17-401d-8529-c02d480d9f25-trusted-ca\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de633ac5-2c17-401d-8529-c02d480d9f25-config\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-default-certificate\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-config\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.191919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191982 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmcq\" (UniqueName: \"kubernetes.io/projected/938f432d-fa76-41c9-b82b-25f85fbb30be-kube-api-access-mvmcq\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faeb92d0-d3b8-4602-95ce-941edc7337dd-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-stats-auth\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192092 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-installation-pull-secrets\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192245 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/faeb92d0-d3b8-4602-95ce-941edc7337dd-snapshots\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.191899 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faeb92d0-d3b8-4602-95ce-941edc7337dd-tmp\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.192434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192375 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faeb92d0-d3b8-4602-95ce-941edc7337dd-service-ca-bundle\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.192874 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192777 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de633ac5-2c17-401d-8529-c02d480d9f25-config\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.192874 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.192800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de633ac5-2c17-401d-8529-c02d480d9f25-trusted-ca\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.194515 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.194492 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/faeb92d0-d3b8-4602-95ce-941edc7337dd-serving-cert\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.194809 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.194772 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-stats-auth\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.194971 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.194946 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de633ac5-2c17-401d-8529-c02d480d9f25-serving-cert\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.195135 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.195119 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-default-certificate\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.200398 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.200351 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2b4l\" (UniqueName: \"kubernetes.io/projected/faeb92d0-d3b8-4602-95ce-941edc7337dd-kube-api-access-t2b4l\") pod \"insights-operator-5885fdb759-s6q2j\" (UID: \"faeb92d0-d3b8-4602-95ce-941edc7337dd\") " pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.201149 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.201123 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tbk\" (UniqueName: \"kubernetes.io/projected/de633ac5-2c17-401d-8529-c02d480d9f25-kube-api-access-96tbk\") pod \"console-operator-584dcb6b54-frwvb\" (UID: \"de633ac5-2c17-401d-8529-c02d480d9f25\") " pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.202989 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.202954 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkpr\" (UniqueName: \"kubernetes.io/projected/d1cdec05-18f6-4d64-ac29-65b77e5ff36b-kube-api-access-hfkpr\") pod \"volume-data-source-validator-549c6d54cf-lgkl2\" (UID: \"d1cdec05-18f6-4d64-ac29-65b77e5ff36b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" Jul 24 19:26:45.203397 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.203374 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vkx\" (UniqueName: \"kubernetes.io/projected/9996735c-0afe-414c-b463-b2f4c223d41c-kube-api-access-b9vkx\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.205125 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.205068 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.206706 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.206685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.207040 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.207018 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6mf\" (UniqueName: \"kubernetes.io/projected/fc62b1cf-7e3a-4df4-9272-82d6e407dfec-kube-api-access-5v6mf\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2\" (UID: \"fc62b1cf-7e3a-4df4-9272-82d6e407dfec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.223163 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.223141 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" Jul 24 19:26:45.242181 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.242146 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-s6q2j" Jul 24 19:26:45.259920 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.259901 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" Jul 24 19:26:45.285878 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.285852 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:45.292612 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-trusted-ca\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.292751 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/938f432d-fa76-41c9-b82b-25f85fbb30be-config-volume\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.292751 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292660 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-image-registry-private-configuration\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.292751 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blsx4\" (UniqueName: \"kubernetes.io/projected/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-kube-api-access-blsx4\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.292751 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-config\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292775 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmcq\" (UniqueName: \"kubernetes.io/projected/938f432d-fa76-41c9-b82b-25f85fbb30be-kube-api-access-mvmcq\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-installation-pull-secrets\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/938f432d-fa76-41c9-b82b-25f85fbb30be-tmp-dir\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-kube-api-access-ltf6j\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9q79\" (UniqueName: \"kubernetes.io/projected/028f41d9-d9c3-44ed-85af-4c7078cf2cd4-kube-api-access-h9q79\") pod \"ingress-canary-mnvj7\" (UID: \"028f41d9-d9c3-44ed-85af-4c7078cf2cd4\") " pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.292962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.292962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgplx\" (UniqueName: \"kubernetes.io/projected/7da57dfc-bfd9-4fa4-a39f-a90957d47706-kube-api-access-tgplx\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/028f41d9-d9c3-44ed-85af-4c7078cf2cd4-cert\") pod \"ingress-canary-mnvj7\" (UID: \"028f41d9-d9c3-44ed-85af-4c7078cf2cd4\") " pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-ca-trust-extracted\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-certificates\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-bound-sa-token\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-serving-cert\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sk9\" (UniqueName: \"kubernetes.io/projected/a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d-kube-api-access-k8sk9\") pod \"network-check-source-7d88cb55d8-wzzdp\" (UID: \"a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293188 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/938f432d-fa76-41c9-b82b-25f85fbb30be-config-volume\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.293274 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 19:26:45.293365 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.293354 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls podName:938f432d-fa76-41c9-b82b-25f85fbb30be nodeName:}" failed. No retries permitted until 2026-07-24 19:26:45.793334625 +0000 UTC m=+35.888823841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls") pod "dns-default-ghxhc" (UID: "938f432d-fa76-41c9-b82b-25f85fbb30be") : secret "dns-default-metrics-tls" not found Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293402 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-config\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.293579 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.293625 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls podName:7da57dfc-bfd9-4fa4-a39f-a90957d47706 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:45.793611322 +0000 UTC m=+35.889100532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls") pod "cluster-samples-operator-6748489656-4mrdx" (UID: "7da57dfc-bfd9-4fa4-a39f-a90957d47706") : secret "samples-operator-tls" not found Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293624 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/938f432d-fa76-41c9-b82b-25f85fbb30be-tmp-dir\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293624 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-trusted-ca\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.293714 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.293725 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f69ccb89-9t2qh: secret "image-registry-tls" not found Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.293769 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls podName:f808fd24-f54d-4cd6-9a39-8c3ad4258d7a nodeName:}" failed. No retries permitted until 2026-07-24 19:26:45.793757709 +0000 UTC m=+35.889246900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls") pod "image-registry-5f69ccb89-9t2qh" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a") : secret "image-registry-tls" not found Jul 24 19:26:45.293871 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293788 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-ca-trust-extracted\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.294266 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.293964 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-certificates\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.295552 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.295527 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-installation-pull-secrets\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.295850 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.295829 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-serving-cert\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.296124 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.296101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-image-registry-private-configuration\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.296364 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.296321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/028f41d9-d9c3-44ed-85af-4c7078cf2cd4-cert\") pod \"ingress-canary-mnvj7\" (UID: \"028f41d9-d9c3-44ed-85af-4c7078cf2cd4\") " pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.301144 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.301115 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blsx4\" (UniqueName: \"kubernetes.io/projected/e2a9c64f-a5aa-4d52-8981-9aaceb773a22-kube-api-access-blsx4\") pod \"service-ca-operator-5d6c75f84f-qblst\" (UID: \"e2a9c64f-a5aa-4d52-8981-9aaceb773a22\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.301340 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.301324 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmcq\" (UniqueName: \"kubernetes.io/projected/938f432d-fa76-41c9-b82b-25f85fbb30be-kube-api-access-mvmcq\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.305884 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.305864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgplx\" (UniqueName: \"kubernetes.io/projected/7da57dfc-bfd9-4fa4-a39f-a90957d47706-kube-api-access-tgplx\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:45.305959 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.305929 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9q79\" (UniqueName: \"kubernetes.io/projected/028f41d9-d9c3-44ed-85af-4c7078cf2cd4-kube-api-access-h9q79\") pod \"ingress-canary-mnvj7\" (UID: \"028f41d9-d9c3-44ed-85af-4c7078cf2cd4\") " pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.313073 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.312688 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-bound-sa-token\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.313073 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.313015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-kube-api-access-ltf6j\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.354259 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.354231 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnvj7" Jul 24 19:26:45.394985 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.394206 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sk9\" (UniqueName: \"kubernetes.io/projected/a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d-kube-api-access-k8sk9\") pod \"network-check-source-7d88cb55d8-wzzdp\" (UID: \"a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" Jul 24 19:26:45.407403 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.407238 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" Jul 24 19:26:45.423109 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.419401 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sk9\" (UniqueName: \"kubernetes.io/projected/a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d-kube-api-access-k8sk9\") pod \"network-check-source-7d88cb55d8-wzzdp\" (UID: \"a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" Jul 24 19:26:45.444915 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.444589 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" Jul 24 19:26:45.504098 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.504010 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2"] Jul 24 19:26:45.516007 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.515966 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-s6q2j"] Jul 24 19:26:45.518519 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.518496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-frwvb"] Jul 24 19:26:45.527118 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.527100 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2"] Jul 24 19:26:45.545680 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.545661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnvj7"] Jul 24 19:26:45.554376 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:45.554351 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc62b1cf_7e3a_4df4_9272_82d6e407dfec.slice/crio-689095f8526bad6cb1a272f8f28553b0f5a4ef049047c12c571f7b144e5747a0 WatchSource:0}: Error finding container 689095f8526bad6cb1a272f8f28553b0f5a4ef049047c12c571f7b144e5747a0: Status 404 returned error can't find the container with id 689095f8526bad6cb1a272f8f28553b0f5a4ef049047c12c571f7b144e5747a0 Jul 24 19:26:45.554997 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:45.554877 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeb92d0_d3b8_4602_95ce_941edc7337dd.slice/crio-50f1f0ad399a817f1555de73f9a02e84dc1144b5d1e1d58d4be3acaf84432349 WatchSource:0}: Error finding container 50f1f0ad399a817f1555de73f9a02e84dc1144b5d1e1d58d4be3acaf84432349: Status 404 returned error can't find the container with id 50f1f0ad399a817f1555de73f9a02e84dc1144b5d1e1d58d4be3acaf84432349 Jul 24 19:26:45.555729 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:45.555656 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde633ac5_2c17_401d_8529_c02d480d9f25.slice/crio-cf678d3b90d454462354797d790c157ea5d879eab495fbdac57dde7457d6af7f WatchSource:0}: Error finding container cf678d3b90d454462354797d790c157ea5d879eab495fbdac57dde7457d6af7f: Status 404 returned error can't find the container with id cf678d3b90d454462354797d790c157ea5d879eab495fbdac57dde7457d6af7f Jul 24 19:26:45.556761 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:45.556706 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cdec05_18f6_4d64_ac29_65b77e5ff36b.slice/crio-c027a752cad343cc2f193f360dd4b589890b68bf2c25efd1b2de0e112494a638 WatchSource:0}: Error finding container c027a752cad343cc2f193f360dd4b589890b68bf2c25efd1b2de0e112494a638: Status 404 returned error can't find the container with id c027a752cad343cc2f193f360dd4b589890b68bf2c25efd1b2de0e112494a638 Jul 24 19:26:45.557435 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:45.557417 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028f41d9_d9c3_44ed_85af_4c7078cf2cd4.slice/crio-20f7e10d951b395a54f9bbf94bc667a2f37b2c3480308bae84d7d1b21fe4c424 WatchSource:0}: Error finding container 20f7e10d951b395a54f9bbf94bc667a2f37b2c3480308bae84d7d1b21fe4c424: Status 404 returned error can't find the container with id 20f7e10d951b395a54f9bbf94bc667a2f37b2c3480308bae84d7d1b21fe4c424 Jul 24 19:26:45.575108 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.575086 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:26:45.577808 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.577665 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 19:26:45.595982 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.595750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:45.595982 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.595901 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:45.595982 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.595951 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls podName:7267d569-0982-4797-af25-32d1752be10b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:46.595934144 +0000 UTC m=+36.691423342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5xb5f" (UID: "7267d569-0982-4797-af25-32d1752be10b") : secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:45.697076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.697045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.697217 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.697092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:45.697432 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.697340 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 19:26:45.697432 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.697413 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:46.697393216 +0000 UTC m=+36.792882404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : secret "router-metrics-certs-default" not found Jul 24 19:26:45.697624 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.697534 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:46.697509949 +0000 UTC m=+36.792999151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : configmap references non-existent config key: service-ca.crt Jul 24 19:26:45.731351 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.731324 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst"] Jul 24 19:26:45.734634 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.734354 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp"] Jul 24 19:26:45.734969 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:45.734931 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a9c64f_a5aa_4d52_8981_9aaceb773a22.slice/crio-4f7ddf0d255f8bb3733bdc7eeb9ccb3301993dfc9daca0ee3f58574ae1cddf3d WatchSource:0}: Error finding container 4f7ddf0d255f8bb3733bdc7eeb9ccb3301993dfc9daca0ee3f58574ae1cddf3d: Status 404 returned error can't find the container with id 4f7ddf0d255f8bb3733bdc7eeb9ccb3301993dfc9daca0ee3f58574ae1cddf3d Jul 24 19:26:45.738009 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:45.737979 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a7f0d7_f1cc_4b1f_82e7_432ba7e5e03d.slice/crio-d07e8fe28e17bd10dbc8981a542bbdbb1cdf1fc8ae5c259757a4300ae86f9477 WatchSource:0}: Error finding container d07e8fe28e17bd10dbc8981a542bbdbb1cdf1fc8ae5c259757a4300ae86f9477: Status 404 returned error can't find the container with id d07e8fe28e17bd10dbc8981a542bbdbb1cdf1fc8ae5c259757a4300ae86f9477 Jul 24 19:26:45.797585 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.797564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:45.797658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.797595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:45.797658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.797634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:45.797749 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.797734 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 19:26:45.797795 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.797786 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls podName:7da57dfc-bfd9-4fa4-a39f-a90957d47706 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:46.797770579 +0000 UTC m=+36.893259769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls") pod "cluster-samples-operator-6748489656-4mrdx" (UID: "7da57dfc-bfd9-4fa4-a39f-a90957d47706") : secret "samples-operator-tls" not found Jul 24 19:26:45.797845 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.797788 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 19:26:45.797845 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.797805 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f69ccb89-9t2qh: secret "image-registry-tls" not found Jul 24 19:26:45.797845 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.797734 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 19:26:45.797845 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.797842 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls podName:f808fd24-f54d-4cd6-9a39-8c3ad4258d7a nodeName:}" failed. No retries permitted until 2026-07-24 19:26:46.797829202 +0000 UTC m=+36.893318387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls") pod "image-registry-5f69ccb89-9t2qh" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a") : secret "image-registry-tls" not found Jul 24 19:26:45.798032 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:45.797880 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls podName:938f432d-fa76-41c9-b82b-25f85fbb30be nodeName:}" failed. No retries permitted until 2026-07-24 19:26:46.797863858 +0000 UTC m=+36.893353064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls") pod "dns-default-ghxhc" (UID: "938f432d-fa76-41c9-b82b-25f85fbb30be") : secret "dns-default-metrics-tls" not found Jul 24 19:26:45.838368 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.838338 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerStarted","Data":"f1a6084977085a45ade3b7269d48dd610b374906eeca04cdf50f336f9dc7aa66"} Jul 24 19:26:45.839506 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.839462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" event={"ID":"e2a9c64f-a5aa-4d52-8981-9aaceb773a22","Type":"ContainerStarted","Data":"4f7ddf0d255f8bb3733bdc7eeb9ccb3301993dfc9daca0ee3f58574ae1cddf3d"} Jul 24 19:26:45.840655 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.840634 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" event={"ID":"a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d","Type":"ContainerStarted","Data":"d07e8fe28e17bd10dbc8981a542bbdbb1cdf1fc8ae5c259757a4300ae86f9477"} Jul 24 19:26:45.842017 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.841991 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" event={"ID":"de633ac5-2c17-401d-8529-c02d480d9f25","Type":"ContainerStarted","Data":"cf678d3b90d454462354797d790c157ea5d879eab495fbdac57dde7457d6af7f"} Jul 24 19:26:45.843129 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.843104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" event={"ID":"d1cdec05-18f6-4d64-ac29-65b77e5ff36b","Type":"ContainerStarted","Data":"c027a752cad343cc2f193f360dd4b589890b68bf2c25efd1b2de0e112494a638"} Jul 24 19:26:45.844226 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.844203 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnvj7" event={"ID":"028f41d9-d9c3-44ed-85af-4c7078cf2cd4","Type":"ContainerStarted","Data":"20f7e10d951b395a54f9bbf94bc667a2f37b2c3480308bae84d7d1b21fe4c424"} Jul 24 19:26:45.845248 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.845224 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-s6q2j" event={"ID":"faeb92d0-d3b8-4602-95ce-941edc7337dd","Type":"ContainerStarted","Data":"50f1f0ad399a817f1555de73f9a02e84dc1144b5d1e1d58d4be3acaf84432349"} Jul 24 19:26:45.846345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.846323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" event={"ID":"fc62b1cf-7e3a-4df4-9272-82d6e407dfec","Type":"ContainerStarted","Data":"689095f8526bad6cb1a272f8f28553b0f5a4ef049047c12c571f7b144e5747a0"} Jul 24 19:26:45.945510 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.945450 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-99pmx"] Jul 24 19:26:45.958217 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.958192 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:45.960780 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.960759 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 19:26:45.960890 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.960781 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 19:26:45.960890 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.960830 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-624qv\"" Jul 24 19:26:45.961190 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:45.961165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-99pmx"] Jul 24 19:26:46.096685 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.096655 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:46.100623 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.100557 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:46.100720 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.100669 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:46.182517 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.182401 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:46.201294 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.201212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:46.201411 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.201315 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:46.201589 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.201562 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 19:26:46.201699 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.201624 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert podName:c246e701-32dc-4ec9-a53d-b4e9cffdff6b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:46.701607988 +0000 UTC m=+36.797097181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-99pmx" (UID: "c246e701-32dc-4ec9-a53d-b4e9cffdff6b") : secret "networking-console-plugin-cert" not found Jul 24 19:26:46.201918 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.201878 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:46.283549 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.283521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:46.454803 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.454588 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:46.576444 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.575387 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:26:46.576444 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.575836 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:26:46.578990 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.578708 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nz88f\"" Jul 24 19:26:46.579408 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.579016 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 19:26:46.579408 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.579220 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ln4l7\"" Jul 24 19:26:46.605568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.604901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:46.605568 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.605143 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:46.605568 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.605206 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls podName:7267d569-0982-4797-af25-32d1752be10b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:48.6051876 +0000 UTC m=+38.700676797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5xb5f" (UID: "7267d569-0982-4797-af25-32d1752be10b") : secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:46.706532 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.705655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:46.706532 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.705712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:46.706532 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.705806 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:46.707579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.706918 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:48.706897634 +0000 UTC m=+38.802386819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : configmap references non-existent config key: service-ca.crt Jul 24 19:26:46.707579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.707336 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 19:26:46.707579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.707382 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert podName:c246e701-32dc-4ec9-a53d-b4e9cffdff6b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:47.707367623 +0000 UTC m=+37.802856811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-99pmx" (UID: "c246e701-32dc-4ec9-a53d-b4e9cffdff6b") : secret "networking-console-plugin-cert" not found Jul 24 19:26:46.707579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.707506 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 19:26:46.707579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.707548 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:48.707534922 +0000 UTC m=+38.803024107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : secret "router-metrics-certs-default" not found Jul 24 19:26:46.806768 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.806734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:46.806938 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.806787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:46.806938 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.806838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:46.807042 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.806988 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 19:26:46.807042 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.807003 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f69ccb89-9t2qh: secret "image-registry-tls" not found Jul 24 19:26:46.807135 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.807055 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls podName:f808fd24-f54d-4cd6-9a39-8c3ad4258d7a nodeName:}" failed. No retries permitted until 2026-07-24 19:26:48.8070382 +0000 UTC m=+38.902527400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls") pod "image-registry-5f69ccb89-9t2qh" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a") : secret "image-registry-tls" not found Jul 24 19:26:46.807454 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.807433 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 19:26:46.807551 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.807506 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls podName:938f432d-fa76-41c9-b82b-25f85fbb30be nodeName:}" failed. No retries permitted until 2026-07-24 19:26:48.807492043 +0000 UTC m=+38.902981241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls") pod "dns-default-ghxhc" (UID: "938f432d-fa76-41c9-b82b-25f85fbb30be") : secret "dns-default-metrics-tls" not found Jul 24 19:26:46.807619 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.807570 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 19:26:46.807619 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:46.807601 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls podName:7da57dfc-bfd9-4fa4-a39f-a90957d47706 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:48.807590821 +0000 UTC m=+38.903080042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls") pod "cluster-samples-operator-6748489656-4mrdx" (UID: "7da57dfc-bfd9-4fa4-a39f-a90957d47706") : secret "samples-operator-tls" not found Jul 24 19:26:46.856295 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.856199 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:46.864909 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.864885 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e4d9366-97b2-4601-89c8-04e78cccb813" containerID="f1a6084977085a45ade3b7269d48dd610b374906eeca04cdf50f336f9dc7aa66" exitCode=0 Jul 24 19:26:46.865075 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.864929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerDied","Data":"f1a6084977085a45ade3b7269d48dd610b374906eeca04cdf50f336f9dc7aa66"} Jul 24 19:26:46.913366 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:46.913257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:47.717497 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:47.717235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:47.717497 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:47.717458 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 19:26:47.717974 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:47.717534 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert podName:c246e701-32dc-4ec9-a53d-b4e9cffdff6b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:49.717514538 +0000 UTC m=+39.813003723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-99pmx" (UID: "c246e701-32dc-4ec9-a53d-b4e9cffdff6b") : secret "networking-console-plugin-cert" not found Jul 24 19:26:47.881546 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:47.880705 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e4d9366-97b2-4601-89c8-04e78cccb813" containerID="531c78391fba87e458e904bc5509ec94d67ba4af1d688639796a640d7907c28b" exitCode=0 Jul 24 19:26:47.881546 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:47.880799 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerDied","Data":"531c78391fba87e458e904bc5509ec94d67ba4af1d688639796a640d7907c28b"} Jul 24 19:26:47.889528 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:47.889270 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" event={"ID":"d1cdec05-18f6-4d64-ac29-65b77e5ff36b","Type":"ContainerStarted","Data":"d8597fe1aa329b844085be387f7df4cd8a710925c4cce8527d0060934da8320b"} Jul 24 19:26:47.922077 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:47.922025 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lgkl2" podStartSLOduration=30.062346424 podStartE2EDuration="31.922007957s" totalImagesPullingTime="1.859661533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:47.920918147 +0000 UTC m=+38.016407352" watchObservedRunningTime="2026-07-24 19:26:47.922007957 +0000 UTC m=+38.017497156" Jul 24 19:26:48.016217 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.016144 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:48.117726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.117665 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:48.627562 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.627526 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:48.627741 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.627694 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:48.627804 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.627775 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls podName:7267d569-0982-4797-af25-32d1752be10b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:52.627754282 +0000 UTC m=+42.723243468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5xb5f" (UID: "7267d569-0982-4797-af25-32d1752be10b") : secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:48.728637 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.728600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:48.728637 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.728633 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:48.729179 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.728745 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 19:26:48.729179 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.728788 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:52.728763822 +0000 UTC m=+42.824253019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : configmap references non-existent config key: service-ca.crt Jul 24 19:26:48.729179 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.728812 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:52.72880375 +0000 UTC m=+42.824292935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : secret "router-metrics-certs-default" not found Jul 24 19:26:48.829661 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.829630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:48.829818 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.829666 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:48.829818 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.829701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:48.829818 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.829802 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 19:26:48.829974 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.829814 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 19:26:48.829974 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.829834 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f69ccb89-9t2qh: secret "image-registry-tls" not found Jul 24 19:26:48.829974 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.829872 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls podName:938f432d-fa76-41c9-b82b-25f85fbb30be nodeName:}" failed. No retries permitted until 2026-07-24 19:26:52.829852788 +0000 UTC m=+42.925341975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls") pod "dns-default-ghxhc" (UID: "938f432d-fa76-41c9-b82b-25f85fbb30be") : secret "dns-default-metrics-tls" not found Jul 24 19:26:48.829974 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.829811 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 19:26:48.829974 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.829892 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls podName:f808fd24-f54d-4cd6-9a39-8c3ad4258d7a nodeName:}" failed. No retries permitted until 2026-07-24 19:26:52.829882773 +0000 UTC m=+42.925371963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls") pod "image-registry-5f69ccb89-9t2qh" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a") : secret "image-registry-tls" not found Jul 24 19:26:48.829974 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:48.829944 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls podName:7da57dfc-bfd9-4fa4-a39f-a90957d47706 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:52.829927295 +0000 UTC m=+42.925416490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls") pod "cluster-samples-operator-6748489656-4mrdx" (UID: "7da57dfc-bfd9-4fa4-a39f-a90957d47706") : secret "samples-operator-tls" not found Jul 24 19:26:48.896051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.895992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cx256" event={"ID":"3e4d9366-97b2-4601-89c8-04e78cccb813","Type":"ContainerStarted","Data":"56ccdf3c243ff2a950196a238f57427a53be7bead30465d11029a7a3cb923a43"} Jul 24 19:26:48.929348 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:48.929301 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cx256" podStartSLOduration=8.532367348 podStartE2EDuration="38.929285479s" totalImagesPullingTime="30.396918131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:48.928506149 +0000 UTC m=+39.023995356" watchObservedRunningTime="2026-07-24 19:26:48.929285479 +0000 UTC m=+39.024774685" Jul 24 19:26:49.077027 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:49.076972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:49.143166 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:49.143129 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:49.599785 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:49.599748 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:49.657726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:49.657692 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:49.739916 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:49.739878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:49.740506 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:49.740030 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 19:26:49.740506 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:49.740115 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert podName:c246e701-32dc-4ec9-a53d-b4e9cffdff6b nodeName:}" failed. No retries permitted until 2026-07-24 19:26:53.740093633 +0000 UTC m=+43.835583017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-99pmx" (UID: "c246e701-32dc-4ec9-a53d-b4e9cffdff6b") : secret "networking-console-plugin-cert" not found Jul 24 19:26:52.607289 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.607252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:52.664483 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.664440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:26:52.664648 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.664587 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:52.664707 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.664668 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls podName:7267d569-0982-4797-af25-32d1752be10b nodeName:}" failed. No retries permitted until 2026-07-24 19:27:00.664645837 +0000 UTC m=+50.760135034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5xb5f" (UID: "7267d569-0982-4797-af25-32d1752be10b") : secret "cluster-monitoring-operator-tls" not found Jul 24 19:26:52.692645 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.692540 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:52.765574 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.765539 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:52.765731 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.765577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:26:52.765731 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.765694 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 19:26:52.765806 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.765733 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:27:00.765711291 +0000 UTC m=+50.861200481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : configmap references non-existent config key: service-ca.crt Jul 24 19:26:52.765806 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.765762 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:27:00.765751383 +0000 UTC m=+50.861240575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : secret "router-metrics-certs-default" not found Jul 24 19:26:52.792020 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.791982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:52.866441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.866366 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:26:52.866619 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.866517 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 19:26:52.866619 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.866536 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f69ccb89-9t2qh: secret "image-registry-tls" not found Jul 24 19:26:52.866619 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.866544 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:26:52.866619 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.866570 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:26:52.866619 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.866596 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls podName:f808fd24-f54d-4cd6-9a39-8c3ad4258d7a nodeName:}" failed. No retries permitted until 2026-07-24 19:27:00.866576345 +0000 UTC m=+50.962065558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls") pod "image-registry-5f69ccb89-9t2qh" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a") : secret "image-registry-tls" not found Jul 24 19:26:52.866875 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.866643 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 19:26:52.866875 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.866693 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 19:26:52.866875 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.866695 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls podName:7da57dfc-bfd9-4fa4-a39f-a90957d47706 nodeName:}" failed. No retries permitted until 2026-07-24 19:27:00.866680895 +0000 UTC m=+50.962170083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls") pod "cluster-samples-operator-6748489656-4mrdx" (UID: "7da57dfc-bfd9-4fa4-a39f-a90957d47706") : secret "samples-operator-tls" not found Jul 24 19:26:52.866875 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:52.866756 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls podName:938f432d-fa76-41c9-b82b-25f85fbb30be nodeName:}" failed. No retries permitted until 2026-07-24 19:27:00.866740464 +0000 UTC m=+50.962229650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls") pod "dns-default-ghxhc" (UID: "938f432d-fa76-41c9-b82b-25f85fbb30be") : secret "dns-default-metrics-tls" not found Jul 24 19:26:52.895214 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.895182 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:52.992896 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:52.992859 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:53.773503 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.773330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:26:53.773839 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:53.773522 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 19:26:53.773839 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:53.773570 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert podName:c246e701-32dc-4ec9-a53d-b4e9cffdff6b nodeName:}" failed. No retries permitted until 2026-07-24 19:27:01.773556309 +0000 UTC m=+51.869045495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-99pmx" (UID: "c246e701-32dc-4ec9-a53d-b4e9cffdff6b") : secret "networking-console-plugin-cert" not found Jul 24 19:26:53.910995 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.910434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" event={"ID":"e2a9c64f-a5aa-4d52-8981-9aaceb773a22","Type":"ContainerStarted","Data":"3acef5cf490488113bdb3d4f6ed639317d6bb890c84248a8f8ab8ec36fcbfd15"} Jul 24 19:26:53.912717 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.912351 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" event={"ID":"a0a7f0d7-f1cc-4b1f-82e7-432ba7e5e03d","Type":"ContainerStarted","Data":"c54ef1a1d2983e2408d86acc48c799879b5daff0774743522a5b4fe4856abbfb"} Jul 24 19:26:53.914402 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.914379 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/0.log" Jul 24 19:26:53.914515 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.914416 2570 generic.go:356] "Generic (PLEG): container finished" podID="de633ac5-2c17-401d-8529-c02d480d9f25" containerID="c491b58fb555cf394857c427ed35e1237caf13c3110c758e84585c4e819f77e6" exitCode=255 Jul 24 19:26:53.914668 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.914622 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" event={"ID":"de633ac5-2c17-401d-8529-c02d480d9f25","Type":"ContainerDied","Data":"c491b58fb555cf394857c427ed35e1237caf13c3110c758e84585c4e819f77e6"} Jul 24 19:26:53.914780 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.914762 2570 scope.go:117] "RemoveContainer" containerID="c491b58fb555cf394857c427ed35e1237caf13c3110c758e84585c4e819f77e6" Jul 24 19:26:53.917082 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.916677 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnvj7" event={"ID":"028f41d9-d9c3-44ed-85af-4c7078cf2cd4","Type":"ContainerStarted","Data":"947ed7e85b22e3b41f800491fb6aa11d29b0786f726b3df181a3921e9d0ea01f"} Jul 24 19:26:53.918644 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.918593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-s6q2j" event={"ID":"faeb92d0-d3b8-4602-95ce-941edc7337dd","Type":"ContainerStarted","Data":"2c28d8b5b84d11edb91368da59403e80c2e67fb378175ed7c8dde22fba5ad252"} Jul 24 19:26:53.920618 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.920264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" event={"ID":"fc62b1cf-7e3a-4df4-9272-82d6e407dfec","Type":"ContainerStarted","Data":"60be0c9f3d230f1c68d3e85e6f0a2c46553b0abbddf0c5cf8eaab7ce59108352"} Jul 24 19:26:53.932543 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.928758 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" podStartSLOduration=29.105298205 podStartE2EDuration="36.928743613s" totalImagesPullingTime="7.823445408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:53.928369668 +0000 UTC m=+44.023858877" watchObservedRunningTime="2026-07-24 19:26:53.928743613 +0000 UTC m=+44.024232809" Jul 24 19:26:53.951702 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.949006 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-s6q2j" podStartSLOduration=29.983819867 podStartE2EDuration="37.948986362s" totalImagesPullingTime="7.965166495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:53.947278983 +0000 UTC m=+44.042768191" watchObservedRunningTime="2026-07-24 19:26:53.948986362 +0000 UTC m=+44.044475548" Jul 24 19:26:53.963977 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:53.963692 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mnvj7" podStartSLOduration=1.9984458699999998 podStartE2EDuration="9.963675148s" totalImagesPullingTime="7.965229278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:53.963448981 +0000 UTC m=+44.058938189" watchObservedRunningTime="2026-07-24 19:26:53.963675148 +0000 UTC m=+44.059164355" Jul 24 19:26:54.001511 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.001450 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" podStartSLOduration=29.996582939 podStartE2EDuration="38.001435121s" totalImagesPullingTime="8.004852182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:54.000550365 +0000 UTC m=+44.096039601" watchObservedRunningTime="2026-07-24 19:26:54.001435121 +0000 UTC m=+44.096924328" Jul 24 19:26:54.018055 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.018005 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzzdp" podStartSLOduration=28.187284922 podStartE2EDuration="36.017991345s" totalImagesPullingTime="7.830706423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:54.016737374 +0000 UTC m=+44.112226582" watchObservedRunningTime="2026-07-24 19:26:54.017991345 +0000 UTC m=+44.113480595" Jul 24 19:26:54.444734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.444651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:54.500628 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.500595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:54.911652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.911593 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h59ph"] Jul 24 19:26:54.926709 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.926676 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:26:54.927133 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.927109 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/0.log" Jul 24 19:26:54.927267 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.927158 2570 generic.go:356] "Generic (PLEG): container finished" podID="de633ac5-2c17-401d-8529-c02d480d9f25" containerID="564086c1f5fc725799cc4b6a6861acd4b4d00328c6728eed12744aae3de53e4e" exitCode=255 Jul 24 19:26:54.928110 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.928088 2570 scope.go:117] "RemoveContainer" containerID="564086c1f5fc725799cc4b6a6861acd4b4d00328c6728eed12744aae3de53e4e" Jul 24 19:26:54.928269 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:54.928240 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-frwvb_openshift-console-operator(de633ac5-2c17-401d-8529-c02d480d9f25)\"" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" podUID="de633ac5-2c17-401d-8529-c02d480d9f25" Jul 24 19:26:54.936418 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.936388 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" event={"ID":"de633ac5-2c17-401d-8529-c02d480d9f25","Type":"ContainerDied","Data":"564086c1f5fc725799cc4b6a6861acd4b4d00328c6728eed12744aae3de53e4e"} Jul 24 19:26:54.936538 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.936432 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h59ph"] Jul 24 19:26:54.936538 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.936484 2570 scope.go:117] "RemoveContainer" containerID="c491b58fb555cf394857c427ed35e1237caf13c3110c758e84585c4e819f77e6" Jul 24 19:26:54.936538 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.936512 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:54.939120 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.939102 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 19:26:54.939367 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.939351 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 19:26:54.939653 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.939638 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qvrwm\"" Jul 24 19:26:54.954387 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.954359 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:54.985201 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.985171 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnkkl\" (UniqueName: \"kubernetes.io/projected/d30c97b9-f517-4a45-896d-5cdf53128137-kube-api-access-qnkkl\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:54.985304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.985246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30c97b9-f517-4a45-896d-5cdf53128137-crio-socket\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:54.985440 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.985417 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30c97b9-f517-4a45-896d-5cdf53128137-data-volume\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:54.985516 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.985461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:54.986151 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:54.985734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.017047 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.017017 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:55.087141 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.087109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.087269 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.087188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkkl\" (UniqueName: \"kubernetes.io/projected/d30c97b9-f517-4a45-896d-5cdf53128137-kube-api-access-qnkkl\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.087269 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.087231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30c97b9-f517-4a45-896d-5cdf53128137-crio-socket\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.087269 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.087236 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 19:26:55.087405 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.087286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30c97b9-f517-4a45-896d-5cdf53128137-data-volume\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.087405 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.087300 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:55.587281684 +0000 UTC m=+45.682770885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : configmap "kube-rbac-proxy" not found Jul 24 19:26:55.087405 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.087335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.087585 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.087518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30c97b9-f517-4a45-896d-5cdf53128137-crio-socket\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.087585 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.087538 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 19:26:55.087585 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.087575 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:55.587563656 +0000 UTC m=+45.683052853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : secret "insights-runtime-extractor-tls" not found Jul 24 19:26:55.087702 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.087614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30c97b9-f517-4a45-896d-5cdf53128137-data-volume\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.097012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.096975 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkkl\" (UniqueName: \"kubernetes.io/projected/d30c97b9-f517-4a45-896d-5cdf53128137-kube-api-access-qnkkl\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.219761 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.219651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:55.279354 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.279316 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:55.286340 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.286314 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:55.286503 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.286384 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:26:55.592291 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.592266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.592579 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.592357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:55.592579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.592395 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 19:26:55.592579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.592460 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:56.592445184 +0000 UTC m=+46.687934384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : configmap "kube-rbac-proxy" not found Jul 24 19:26:55.592579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.592508 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 19:26:55.592579 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.592560 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:56.592546188 +0000 UTC m=+46.688035372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : secret "insights-runtime-extractor-tls" not found Jul 24 19:26:55.845055 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.844963 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:55.933599 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.933578 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:26:55.933966 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.933878 2570 scope.go:117] "RemoveContainer" containerID="564086c1f5fc725799cc4b6a6861acd4b4d00328c6728eed12744aae3de53e4e" Jul 24 19:26:55.934032 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:55.934014 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-frwvb_openshift-console-operator(de633ac5-2c17-401d-8529-c02d480d9f25)\"" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" podUID="de633ac5-2c17-401d-8529-c02d480d9f25" Jul 24 19:26:55.944800 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:55.944772 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:56.119485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.119391 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j425q_54646590-a397-4aab-8b6f-a4de53ca338d/dns-node-resolver/0.log" Jul 24 19:26:56.132368 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.132344 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:56.164385 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.164358 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6"] Jul 24 19:26:56.197780 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.197756 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6"] Jul 24 19:26:56.197912 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.197858 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" Jul 24 19:26:56.200313 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.200290 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 19:26:56.200741 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.200723 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 19:26:56.200828 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.200748 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bzw8c\"" Jul 24 19:26:56.217229 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.217195 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:56.270665 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.270637 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:56.297321 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.297296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdw5r\" (UniqueName: \"kubernetes.io/projected/fb7155bf-ad1d-42c4-b91d-b036ef58de8f-kube-api-access-xdw5r\") pod \"migrator-6865c9967-9txg6\" (UID: \"fb7155bf-ad1d-42c4-b91d-b036ef58de8f\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" Jul 24 19:26:56.398004 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.397943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdw5r\" (UniqueName: \"kubernetes.io/projected/fb7155bf-ad1d-42c4-b91d-b036ef58de8f-kube-api-access-xdw5r\") pod \"migrator-6865c9967-9txg6\" (UID: \"fb7155bf-ad1d-42c4-b91d-b036ef58de8f\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" Jul 24 19:26:56.409482 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.409438 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdw5r\" (UniqueName: \"kubernetes.io/projected/fb7155bf-ad1d-42c4-b91d-b036ef58de8f-kube-api-access-xdw5r\") pod \"migrator-6865c9967-9txg6\" (UID: \"fb7155bf-ad1d-42c4-b91d-b036ef58de8f\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" Jul 24 19:26:56.507427 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.507394 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" Jul 24 19:26:56.598970 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.598910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:56.599149 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:56.599052 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 19:26:56.599149 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.599057 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:56.599149 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:56.599107 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:58.599091772 +0000 UTC m=+48.694580970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : configmap "kube-rbac-proxy" not found Jul 24 19:26:56.599375 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:56.599177 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 19:26:56.599375 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:56.599229 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:26:58.599214839 +0000 UTC m=+48.694704030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : secret "insights-runtime-extractor-tls" not found Jul 24 19:26:56.653496 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.653454 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6"] Jul 24 19:26:56.656136 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:56.656106 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7155bf_ad1d_42c4_b91d_b036ef58de8f.slice/crio-12a504465d3e7a0e7657ca986851ae690332f4881cf670fd5ca54c3e8af56461 WatchSource:0}: Error finding container 12a504465d3e7a0e7657ca986851ae690332f4881cf670fd5ca54c3e8af56461: Status 404 returned error can't find the container with id 12a504465d3e7a0e7657ca986851ae690332f4881cf670fd5ca54c3e8af56461 Jul 24 19:26:56.937841 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.937762 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" event={"ID":"fb7155bf-ad1d-42c4-b91d-b036ef58de8f","Type":"ContainerStarted","Data":"12a504465d3e7a0e7657ca986851ae690332f4881cf670fd5ca54c3e8af56461"} Jul 24 19:26:56.938166 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:56.938046 2570 scope.go:117] "RemoveContainer" containerID="564086c1f5fc725799cc4b6a6861acd4b4d00328c6728eed12744aae3de53e4e" Jul 24 19:26:56.938222 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:56.938208 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-frwvb_openshift-console-operator(de633ac5-2c17-401d-8529-c02d480d9f25)\"" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" podUID="de633ac5-2c17-401d-8529-c02d480d9f25" Jul 24 19:26:57.315619 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.315593 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqltr_4a2b5609-a8ac-4be7-a8d6-91bce606bfd1/node-ca/0.log" Jul 24 19:26:57.443399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.443369 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8lcsc"] Jul 24 19:26:57.458271 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.458115 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8lcsc"] Jul 24 19:26:57.458271 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.458235 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.461524 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.461498 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 19:26:57.462984 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.462961 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 19:26:57.463099 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.463007 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 19:26:57.463800 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.463782 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 19:26:57.463934 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.463917 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9lxbn\"" Jul 24 19:26:57.483825 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.480637 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:57.507306 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.507279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05af9904-3837-4a85-ae6c-e04668888dde-signing-cabundle\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.507544 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.507524 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzhz\" (UniqueName: \"kubernetes.io/projected/05af9904-3837-4a85-ae6c-e04668888dde-kube-api-access-9nzhz\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.507610 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.507572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05af9904-3837-4a85-ae6c-e04668888dde-signing-key\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.570988 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.570923 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:57.608576 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.608551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzhz\" (UniqueName: \"kubernetes.io/projected/05af9904-3837-4a85-ae6c-e04668888dde-kube-api-access-9nzhz\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.608688 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.608612 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05af9904-3837-4a85-ae6c-e04668888dde-signing-key\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.608688 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.608653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05af9904-3837-4a85-ae6c-e04668888dde-signing-cabundle\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.609385 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.609362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05af9904-3837-4a85-ae6c-e04668888dde-signing-cabundle\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.611245 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.611224 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05af9904-3837-4a85-ae6c-e04668888dde-signing-key\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.618433 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.618413 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzhz\" (UniqueName: \"kubernetes.io/projected/05af9904-3837-4a85-ae6c-e04668888dde-kube-api-access-9nzhz\") pod \"service-ca-658d7b7c78-8lcsc\" (UID: \"05af9904-3837-4a85-ae6c-e04668888dde\") " pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.716047 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.716020 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mnvj7_028f41d9-d9c3-44ed-85af-4c7078cf2cd4/serve-healthcheck-canary/0.log" Jul 24 19:26:57.769841 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.769817 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" Jul 24 19:26:57.919180 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.919150 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8lcsc"] Jul 24 19:26:57.922786 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:26:57.922762 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05af9904_3837_4a85_ae6c_e04668888dde.slice/crio-d0ab6bc0afa5750d4a5a5b2f60276082c6fb274883f1f53f32133f63320e82ac WatchSource:0}: Error finding container d0ab6bc0afa5750d4a5a5b2f60276082c6fb274883f1f53f32133f63320e82ac: Status 404 returned error can't find the container with id d0ab6bc0afa5750d4a5a5b2f60276082c6fb274883f1f53f32133f63320e82ac Jul 24 19:26:57.942381 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:57.942355 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" event={"ID":"05af9904-3837-4a85-ae6c-e04668888dde","Type":"ContainerStarted","Data":"d0ab6bc0afa5750d4a5a5b2f60276082c6fb274883f1f53f32133f63320e82ac"} Jul 24 19:26:58.281048 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.281018 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:58.381979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.381941 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:58.488888 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.488844 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:58.617779 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.617758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:58.617877 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.617840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:26:58.617932 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:58.617882 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 19:26:58.617932 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:58.617931 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:27:02.617916722 +0000 UTC m=+52.713405908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : secret "insights-runtime-extractor-tls" not found Jul 24 19:26:58.618045 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:58.617974 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 19:26:58.618089 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:26:58.618049 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:27:02.618030072 +0000 UTC m=+52.713519277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : configmap "kube-rbac-proxy" not found Jul 24 19:26:58.947459 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.947419 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" event={"ID":"fb7155bf-ad1d-42c4-b91d-b036ef58de8f","Type":"ContainerStarted","Data":"58f80d38877aded57e0da0fee38bcaf70c1336c908e5167fb0f258dd18d10350"} Jul 24 19:26:58.947849 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.947503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" event={"ID":"fb7155bf-ad1d-42c4-b91d-b036ef58de8f","Type":"ContainerStarted","Data":"7281590f2e2009ffd0bcf6a595389b310409b44f86f9c36fb41dbbac2c4a7e3b"} Jul 24 19:26:58.948885 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.948863 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" event={"ID":"05af9904-3837-4a85-ae6c-e04668888dde","Type":"ContainerStarted","Data":"627a7112b0e190b811c98b4f881800635a816c9d55a8b87de61164666a22704c"} Jul 24 19:26:58.968765 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.968722 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9txg6" podStartSLOduration=1.012885222 podStartE2EDuration="2.968704598s" totalImagesPullingTime="1.955819376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:58.967603946 +0000 UTC m=+49.063093153" watchObservedRunningTime="2026-07-24 19:26:58.968704598 +0000 UTC m=+49.064193807" Jul 24 19:26:58.986611 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:58.986570 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-8lcsc" podStartSLOduration=1.9865557109999998 podStartE2EDuration="1.986555711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:58.986376772 +0000 UTC m=+49.081865978" watchObservedRunningTime="2026-07-24 19:26:58.986555711 +0000 UTC m=+49.082044920" Jul 24 19:26:59.381202 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:59.381165 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:59.608585 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:59.608537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:59.709015 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:59.708906 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:59.789417 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:59.789372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:26:59.862814 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:26:59.862789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:00.091440 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.091409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:00.546760 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.546729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:00.735673 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.735638 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:27:00.735829 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.735791 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 19:27:00.735870 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.735855 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls podName:7267d569-0982-4797-af25-32d1752be10b nodeName:}" failed. No retries permitted until 2026-07-24 19:27:16.735839463 +0000 UTC m=+66.831328648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5xb5f" (UID: "7267d569-0982-4797-af25-32d1752be10b") : secret "cluster-monitoring-operator-tls" not found Jul 24 19:27:00.836347 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.836269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:00.836347 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.836303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:00.836546 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.836434 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 19:27:00.836546 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.836440 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:27:16.836420908 +0000 UTC m=+66.931910096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : configmap references non-existent config key: service-ca.crt Jul 24 19:27:00.836546 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.836488 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs podName:9996735c-0afe-414c-b463-b2f4c223d41c nodeName:}" failed. No retries permitted until 2026-07-24 19:27:16.83645922 +0000 UTC m=+66.931948404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs") pod "router-default-56947db777-2bfh7" (UID: "9996735c-0afe-414c-b463-b2f4c223d41c") : secret "router-metrics-certs-default" not found Jul 24 19:27:00.936651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.936627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:27:00.936651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.936657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:27:00.936856 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.936694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:27:00.936856 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.936773 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 19:27:00.936856 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.936817 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 19:27:00.936856 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.936822 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls podName:938f432d-fa76-41c9-b82b-25f85fbb30be nodeName:}" failed. No retries permitted until 2026-07-24 19:27:16.936810102 +0000 UTC m=+67.032299290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls") pod "dns-default-ghxhc" (UID: "938f432d-fa76-41c9-b82b-25f85fbb30be") : secret "dns-default-metrics-tls" not found Jul 24 19:27:00.936856 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.936829 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f69ccb89-9t2qh: secret "image-registry-tls" not found Jul 24 19:27:00.936856 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:00.936860 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls podName:f808fd24-f54d-4cd6-9a39-8c3ad4258d7a nodeName:}" failed. No retries permitted until 2026-07-24 19:27:16.936851324 +0000 UTC m=+67.032340537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls") pod "image-registry-5f69ccb89-9t2qh" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a") : secret "image-registry-tls" not found Jul 24 19:27:00.939080 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.939056 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7da57dfc-bfd9-4fa4-a39f-a90957d47706-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4mrdx\" (UID: \"7da57dfc-bfd9-4fa4-a39f-a90957d47706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:27:00.966935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:00.966916 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" Jul 24 19:27:01.081588 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.081548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:01.101301 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.101229 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx"] Jul 24 19:27:01.158433 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.158392 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:01.480568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.480456 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:01.680729 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.680656 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:01.846484 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.846440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:27:01.846655 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:01.846580 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 19:27:01.846655 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:01.846640 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert podName:c246e701-32dc-4ec9-a53d-b4e9cffdff6b nodeName:}" failed. No retries permitted until 2026-07-24 19:27:17.84662593 +0000 UTC m=+67.942115115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-99pmx" (UID: "c246e701-32dc-4ec9-a53d-b4e9cffdff6b") : secret "networking-console-plugin-cert" not found Jul 24 19:27:01.883838 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.883806 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:01.959385 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:01.959358 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" event={"ID":"7da57dfc-bfd9-4fa4-a39f-a90957d47706","Type":"ContainerStarted","Data":"0a20db385b32b01e7624cb9e860aa8be45975499f5bbb9b8070ac6a1d678b4b9"} Jul 24 19:27:02.088334 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:02.088299 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:02.654248 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:02.654210 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:27:02.654707 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:02.654326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:27:02.654707 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:02.654384 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 19:27:02.654707 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:27:02.654452 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls podName:d30c97b9-f517-4a45-896d-5cdf53128137 nodeName:}" failed. No retries permitted until 2026-07-24 19:27:10.65443541 +0000 UTC m=+60.749924616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h59ph" (UID: "d30c97b9-f517-4a45-896d-5cdf53128137") : secret "insights-runtime-extractor-tls" not found Jul 24 19:27:02.654872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:02.654859 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30c97b9-f517-4a45-896d-5cdf53128137-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:27:02.886722 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:02.886682 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:03.089077 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:03.088893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:03.404524 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:03.404426 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:03.505352 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:03.505317 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:03.680080 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:03.679984 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:03.968759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:03.968679 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" event={"ID":"7da57dfc-bfd9-4fa4-a39f-a90957d47706","Type":"ContainerStarted","Data":"3791f5841bc6e527dd962eaf7f3680437ddab5c2b82ebb8926cdd5a92e28f084"} Jul 24 19:27:03.968759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:03.968737 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" event={"ID":"7da57dfc-bfd9-4fa4-a39f-a90957d47706","Type":"ContainerStarted","Data":"2f2a941ebe7289b43526d35f0f14751cfa7fd1287b2579dd2bb5dfebb145b31b"} Jul 24 19:27:03.985906 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:03.985849 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4mrdx" podStartSLOduration=46.153671787 podStartE2EDuration="47.985830507s" totalImagesPullingTime="1.83215872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:03.985458568 +0000 UTC m=+54.080947778" watchObservedRunningTime="2026-07-24 19:27:03.985830507 +0000 UTC m=+54.081319715" Jul 24 19:27:04.092001 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:04.091968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:04.279340 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:04.279303 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:04.380199 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:04.380122 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:04.481123 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:04.481092 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:05.683084 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:05.683044 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:05.881960 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:05.881929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:06.091047 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:06.091019 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:06.282227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:06.282182 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:06.483193 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:06.483099 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:06.717846 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:06.717812 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:07.707025 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:07.706997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:07.877234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:07.877196 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:08.078121 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:08.078088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:08.285042 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:08.284995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:08.477244 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:08.477162 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:08.831908 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:08.831879 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdlb8" Jul 24 19:27:09.478349 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:09.478320 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:09.588165 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:09.588133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:09.675573 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:09.675547 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:09.881606 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:09.881571 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:10.078895 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.078862 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:10.576848 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.576815 2570 scope.go:117] "RemoveContainer" containerID="564086c1f5fc725799cc4b6a6861acd4b4d00328c6728eed12744aae3de53e4e" Jul 24 19:27:10.724376 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.724338 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:27:10.726673 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.726647 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30c97b9-f517-4a45-896d-5cdf53128137-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h59ph\" (UID: \"d30c97b9-f517-4a45-896d-5cdf53128137\") " pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:27:10.850550 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.850484 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qvrwm\"" Jul 24 19:27:10.858413 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.858394 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h59ph" Jul 24 19:27:10.981880 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.981851 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h59ph"] Jul 24 19:27:10.985053 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:10.985024 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30c97b9_f517_4a45_896d_5cdf53128137.slice/crio-45ef0b6a8d7511b88596a08a3e4a394c1742c9c611398097b2a0da8a2bd2b26a WatchSource:0}: Error finding container 45ef0b6a8d7511b88596a08a3e4a394c1742c9c611398097b2a0da8a2bd2b26a: Status 404 returned error can't find the container with id 45ef0b6a8d7511b88596a08a3e4a394c1742c9c611398097b2a0da8a2bd2b26a Jul 24 19:27:10.991067 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.991049 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:27:10.991159 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.991132 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" event={"ID":"de633ac5-2c17-401d-8529-c02d480d9f25","Type":"ContainerStarted","Data":"f7539a9ce724ae2df8c74f86aa112bbf2c6f7dbe6750081c43277503786c1d1b"} Jul 24 19:27:10.991615 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.991594 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:27:10.992586 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:10.992562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h59ph" event={"ID":"d30c97b9-f517-4a45-896d-5cdf53128137","Type":"ContainerStarted","Data":"45ef0b6a8d7511b88596a08a3e4a394c1742c9c611398097b2a0da8a2bd2b26a"} Jul 24 19:27:11.008049 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:11.008005 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" podStartSLOduration=47.042460172 podStartE2EDuration="55.007992585s" totalImagesPullingTime="7.965532413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:11.007606149 +0000 UTC m=+61.103095355" watchObservedRunningTime="2026-07-24 19:27:11.007992585 +0000 UTC m=+61.103481791" Jul 24 19:27:11.133810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:11.133736 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-frwvb" Jul 24 19:27:11.311836 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:11.311802 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:11.412569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:11.412486 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:11.466348 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:11.466276 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:11.617308 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:11.617266 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:11.994263 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:11.994224 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:12.193559 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:12.193517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:12.250343 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:12.250261 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:12.381769 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:12.381732 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:12.484138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:12.484098 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:12.998139 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:12.998102 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:13.129028 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:13.128956 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:13.231117 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:13.231076 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:13.331278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:13.331204 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:13.432332 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:13.432298 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:13.852728 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:13.852695 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:13.909645 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:13.909603 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:15.164390 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.164357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:27:15.166838 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.166813 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64kz\" (UniqueName: \"kubernetes.io/projected/892c6baa-4dcb-4cda-aeb7-33dc278526a6-kube-api-access-f64kz\") pod \"network-check-target-rhks5\" (UID: \"892c6baa-4dcb-4cda-aeb7-33dc278526a6\") " pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:27:15.265407 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.265374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:27:15.265555 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.265411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:27:15.268050 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.268032 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 19:27:15.268142 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.268048 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 19:27:15.278094 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.278067 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dc9231-d8aa-465a-aa18-a1e9e46cbd18-metrics-certs\") pod \"network-metrics-daemon-6l22f\" (UID: \"06dc9231-d8aa-465a-aa18-a1e9e46cbd18\") " pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:27:15.278094 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.278075 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8132dea3-f97b-44d0-b452-bed78f859762-original-pull-secret\") pod \"global-pull-secret-syncer-plptd\" (UID: \"8132dea3-f97b-44d0-b452-bed78f859762\") " pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:27:15.284119 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.284100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plptd" Jul 24 19:27:15.392147 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.392128 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ln4l7\"" Jul 24 19:27:15.400376 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.400351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l22f" Jul 24 19:27:15.400600 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.400586 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nz88f\"" Jul 24 19:27:15.408583 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:15.408569 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:27:16.243532 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.243496 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t"] Jul 24 19:27:16.260185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.260153 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g"] Jul 24 19:27:16.260355 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.260319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.264362 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.264327 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 19:27:16.264514 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.264385 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 19:27:16.265872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.265274 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 19:27:16.265872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.265408 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 19:27:16.265872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.265417 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 19:27:16.265872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.265540 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 19:27:16.265872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.265719 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 19:27:16.274111 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.274088 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn"] Jul 24 19:27:16.274314 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.274295 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.279999 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.279981 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f762x\"" Jul 24 19:27:16.280151 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.280128 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 19:27:16.292176 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.292157 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t"] Jul 24 19:27:16.292323 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.292193 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g"] Jul 24 19:27:16.292323 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.292281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.294415 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.294377 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn"] Jul 24 19:27:16.294934 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.294767 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 19:27:16.310403 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.310381 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-xrzx2"] Jul 24 19:27:16.334418 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.334400 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xrzx2"] Jul 24 19:27:16.334522 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.334499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xrzx2" Jul 24 19:27:16.336639 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.336621 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 19:27:16.336832 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.336812 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 19:27:16.337236 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.337221 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-74dbw\"" Jul 24 19:27:16.361224 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.361200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:16.374254 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9fc\" (UniqueName: \"kubernetes.io/projected/c6facd7f-2d0b-4779-b131-5cab24f7337e-kube-api-access-2g9fc\") pod \"managed-serviceaccount-addon-agent-6c96bf598b-xjm7g\" (UID: \"c6facd7f-2d0b-4779-b131-5cab24f7337e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.374346 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374295 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-ca\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.374346 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/618ce004-bf5d-4559-8049-ddc31cb8a3af-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.374450 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374385 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vm6f\" (UniqueName: \"kubernetes.io/projected/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-kube-api-access-8vm6f\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.374450 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374437 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.374652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9j75\" (UniqueName: \"kubernetes.io/projected/618ce004-bf5d-4559-8049-ddc31cb8a3af-kube-api-access-t9j75\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.374652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374551 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-tmp\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.374652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374574 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-hub\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.374652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6facd7f-2d0b-4779-b131-5cab24f7337e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c96bf598b-xjm7g\" (UID: \"c6facd7f-2d0b-4779-b131-5cab24f7337e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.374652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.374851 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.374725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6facd7f-2d0b-4779-b131-5cab24f7337e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c96bf598b-xjm7g\" (UID: \"c6facd7f-2d0b-4779-b131-5cab24f7337e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478078 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdr8c\" (UniqueName: \"kubernetes.io/projected/d0267eb1-f96c-4d53-8651-438bb2fbcd4e-kube-api-access-hdr8c\") pod \"downloads-7c97dc4555-xrzx2\" (UID: \"d0267eb1-f96c-4d53-8651-438bb2fbcd4e\") " pod="openshift-console/downloads-7c97dc4555-xrzx2" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9fc\" (UniqueName: \"kubernetes.io/projected/c6facd7f-2d0b-4779-b131-5cab24f7337e-kube-api-access-2g9fc\") pod \"managed-serviceaccount-addon-agent-6c96bf598b-xjm7g\" (UID: \"c6facd7f-2d0b-4779-b131-5cab24f7337e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-ca\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478359 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/618ce004-bf5d-4559-8049-ddc31cb8a3af-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vm6f\" (UniqueName: \"kubernetes.io/projected/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-kube-api-access-8vm6f\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478522 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9j75\" (UniqueName: \"kubernetes.io/projected/618ce004-bf5d-4559-8049-ddc31cb8a3af-kube-api-access-t9j75\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-tmp\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.479031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.478576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-hub\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.483945 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.483884 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-tmp\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.484130 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.484074 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-hub\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.484460 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.484357 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/618ce004-bf5d-4559-8049-ddc31cb8a3af-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.485850 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.485805 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.485953 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.485897 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.486665 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.486638 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6facd7f-2d0b-4779-b131-5cab24f7337e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c96bf598b-xjm7g\" (UID: \"c6facd7f-2d0b-4779-b131-5cab24f7337e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.486802 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.486780 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-ca\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.489118 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.489074 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/618ce004-bf5d-4559-8049-ddc31cb8a3af-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.490038 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.490015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vm6f\" (UniqueName: \"kubernetes.io/projected/87e9f50d-5d1f-4d14-a2a4-9de20d277b1d-kube-api-access-8vm6f\") pod \"klusterlet-addon-workmgr-7b67b7895b-lt9sn\" (UID: \"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.496515 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.496447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9j75\" (UniqueName: \"kubernetes.io/projected/618ce004-bf5d-4559-8049-ddc31cb8a3af-kube-api-access-t9j75\") pod \"cluster-proxy-proxy-agent-5d9f775cb7-n9s7t\" (UID: \"618ce004-bf5d-4559-8049-ddc31cb8a3af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.497418 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.497395 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9fc\" (UniqueName: \"kubernetes.io/projected/c6facd7f-2d0b-4779-b131-5cab24f7337e-kube-api-access-2g9fc\") pod \"managed-serviceaccount-addon-agent-6c96bf598b-xjm7g\" (UID: \"c6facd7f-2d0b-4779-b131-5cab24f7337e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.552730 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.552604 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6l22f"] Jul 24 19:27:16.554789 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:16.554759 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dc9231_d8aa_465a_aa18_a1e9e46cbd18.slice/crio-ef04d6ab8b8ba783b8fdfd3d87499bd2ea7c44d16c7f43a5fdaa307b27e2773b WatchSource:0}: Error finding container ef04d6ab8b8ba783b8fdfd3d87499bd2ea7c44d16c7f43a5fdaa307b27e2773b: Status 404 returned error can't find the container with id ef04d6ab8b8ba783b8fdfd3d87499bd2ea7c44d16c7f43a5fdaa307b27e2773b Jul 24 19:27:16.558345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.558318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:16.579220 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.579197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdr8c\" (UniqueName: \"kubernetes.io/projected/d0267eb1-f96c-4d53-8651-438bb2fbcd4e-kube-api-access-hdr8c\") pod \"downloads-7c97dc4555-xrzx2\" (UID: \"d0267eb1-f96c-4d53-8651-438bb2fbcd4e\") " pod="openshift-console/downloads-7c97dc4555-xrzx2" Jul 24 19:27:16.582146 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.582130 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" Jul 24 19:27:16.591557 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.591532 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" Jul 24 19:27:16.593971 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.593949 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdr8c\" (UniqueName: \"kubernetes.io/projected/d0267eb1-f96c-4d53-8651-438bb2fbcd4e-kube-api-access-hdr8c\") pod \"downloads-7c97dc4555-xrzx2\" (UID: \"d0267eb1-f96c-4d53-8651-438bb2fbcd4e\") " pod="openshift-console/downloads-7c97dc4555-xrzx2" Jul 24 19:27:16.607759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.607736 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:16.645148 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.645054 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xrzx2" Jul 24 19:27:16.672632 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.672541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:16.771716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.771509 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t"] Jul 24 19:27:16.774940 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:16.774902 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod618ce004_bf5d_4559_8049_ddc31cb8a3af.slice/crio-f73b012854241fad7b03a50900d61b1cb1cfb4ee9176908321822a556c37c08b WatchSource:0}: Error finding container f73b012854241fad7b03a50900d61b1cb1cfb4ee9176908321822a556c37c08b: Status 404 returned error can't find the container with id f73b012854241fad7b03a50900d61b1cb1cfb4ee9176908321822a556c37c08b Jul 24 19:27:16.779193 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.779169 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g"] Jul 24 19:27:16.781413 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.781387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:27:16.781595 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:16.781417 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6facd7f_2d0b_4779_b131_5cab24f7337e.slice/crio-73c5b9508573b2bad3e47d8798b826a9f6c1be618aa5d15e16ca0688c02c0d88 WatchSource:0}: Error finding container 73c5b9508573b2bad3e47d8798b826a9f6c1be618aa5d15e16ca0688c02c0d88: Status 404 returned error can't find the container with id 73c5b9508573b2bad3e47d8798b826a9f6c1be618aa5d15e16ca0688c02c0d88 Jul 24 19:27:16.784003 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.783931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7267d569-0982-4797-af25-32d1752be10b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5xb5f\" (UID: \"7267d569-0982-4797-af25-32d1752be10b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:27:16.791084 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.790758 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plptd"] Jul 24 19:27:16.792377 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.792329 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:16.792540 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.792522 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rhks5"] Jul 24 19:27:16.793290 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:16.793265 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892c6baa_4dcb_4cda_aeb7_33dc278526a6.slice/crio-87929946505a5ca3b21768ff9156b876df130747eda94725ca995eaedb4c9748 WatchSource:0}: Error finding container 87929946505a5ca3b21768ff9156b876df130747eda94725ca995eaedb4c9748: Status 404 returned error can't find the container with id 87929946505a5ca3b21768ff9156b876df130747eda94725ca995eaedb4c9748 Jul 24 19:27:16.793394 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.793379 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn"] Jul 24 19:27:16.794407 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:16.794375 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8132dea3_f97b_44d0_b452_bed78f859762.slice/crio-b60116b65c3b8925474f542fbf7b4022be1799e256098c43fbb6a92f7078ff34 WatchSource:0}: Error finding container b60116b65c3b8925474f542fbf7b4022be1799e256098c43fbb6a92f7078ff34: Status 404 returned error can't find the container with id b60116b65c3b8925474f542fbf7b4022be1799e256098c43fbb6a92f7078ff34 Jul 24 19:27:16.795688 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:16.795667 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e9f50d_5d1f_4d14_a2a4_9de20d277b1d.slice/crio-127ab6e78f18f1c2dbccafb63450172c7db0fbce49a54b4db1491c3a6da4936d WatchSource:0}: Error finding container 127ab6e78f18f1c2dbccafb63450172c7db0fbce49a54b4db1491c3a6da4936d: Status 404 returned error can't find the container with id 127ab6e78f18f1c2dbccafb63450172c7db0fbce49a54b4db1491c3a6da4936d Jul 24 19:27:16.837046 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.837001 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xrzx2"] Jul 24 19:27:16.839590 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:16.839565 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0267eb1_f96c_4d53_8651_438bb2fbcd4e.slice/crio-e475103b7056ce907f80ce8f933c84001bb6d50d0fda472b4a11666c1cf306cf WatchSource:0}: Error finding container e475103b7056ce907f80ce8f933c84001bb6d50d0fda472b4a11666c1cf306cf: Status 404 returned error can't find the container with id e475103b7056ce907f80ce8f933c84001bb6d50d0fda472b4a11666c1cf306cf Jul 24 19:27:16.882840 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.882818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:16.882936 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.882849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:16.883446 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.883430 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9996735c-0afe-414c-b463-b2f4c223d41c-service-ca-bundle\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:16.884802 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.884782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9996735c-0afe-414c-b463-b2f4c223d41c-metrics-certs\") pod \"router-default-56947db777-2bfh7\" (UID: \"9996735c-0afe-414c-b463-b2f4c223d41c\") " pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:16.889758 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.889738 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:16.983681 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.983656 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:27:16.983793 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.983711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:27:16.985714 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.985694 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/938f432d-fa76-41c9-b82b-25f85fbb30be-metrics-tls\") pod \"dns-default-ghxhc\" (UID: \"938f432d-fa76-41c9-b82b-25f85fbb30be\") " pod="openshift-dns/dns-default-ghxhc" Jul 24 19:27:16.985889 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:16.985872 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"image-registry-5f69ccb89-9t2qh\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:27:17.017167 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.017148 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hgmb7\"" Jul 24 19:27:17.022903 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.022851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" event={"ID":"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d","Type":"ContainerStarted","Data":"127ab6e78f18f1c2dbccafb63450172c7db0fbce49a54b4db1491c3a6da4936d"} Jul 24 19:27:17.024453 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.024406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rhks5" event={"ID":"892c6baa-4dcb-4cda-aeb7-33dc278526a6","Type":"ContainerStarted","Data":"ab565df9e64c68402a447143badce3d7035f35f69eca872c63ddf7b451c47a02"} Jul 24 19:27:17.024453 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.024445 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rhks5" event={"ID":"892c6baa-4dcb-4cda-aeb7-33dc278526a6","Type":"ContainerStarted","Data":"87929946505a5ca3b21768ff9156b876df130747eda94725ca995eaedb4c9748"} Jul 24 19:27:17.024673 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.024653 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:27:17.024792 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.024772 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" Jul 24 19:27:17.026167 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.026145 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h59ph" event={"ID":"d30c97b9-f517-4a45-896d-5cdf53128137","Type":"ContainerStarted","Data":"d918a0b5868adf75b94e272c480fe481e2696b16b03638537c8f66e26894eb3f"} Jul 24 19:27:17.027483 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.027445 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6l22f" event={"ID":"06dc9231-d8aa-465a-aa18-a1e9e46cbd18","Type":"ContainerStarted","Data":"ef04d6ab8b8ba783b8fdfd3d87499bd2ea7c44d16c7f43a5fdaa307b27e2773b"} Jul 24 19:27:17.028661 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.028638 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plptd" event={"ID":"8132dea3-f97b-44d0-b452-bed78f859762","Type":"ContainerStarted","Data":"b60116b65c3b8925474f542fbf7b4022be1799e256098c43fbb6a92f7078ff34"} Jul 24 19:27:17.029905 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.029883 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" event={"ID":"c6facd7f-2d0b-4779-b131-5cab24f7337e","Type":"ContainerStarted","Data":"73c5b9508573b2bad3e47d8798b826a9f6c1be618aa5d15e16ca0688c02c0d88"} Jul 24 19:27:17.031037 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.030995 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" event={"ID":"618ce004-bf5d-4559-8049-ddc31cb8a3af","Type":"ContainerStarted","Data":"f73b012854241fad7b03a50900d61b1cb1cfb4ee9176908321822a556c37c08b"} Jul 24 19:27:17.032237 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.032213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xrzx2" event={"ID":"d0267eb1-f96c-4d53-8651-438bb2fbcd4e","Type":"ContainerStarted","Data":"e475103b7056ce907f80ce8f933c84001bb6d50d0fda472b4a11666c1cf306cf"} Jul 24 19:27:17.109833 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.109811 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n8wrr\"" Jul 24 19:27:17.117856 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.117771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:17.131123 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.130936 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jcdcc\"" Jul 24 19:27:17.138496 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.138404 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:27:17.182949 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.181429 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rhks5" podStartSLOduration=67.181406074 podStartE2EDuration="1m7.181406074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:17.045878499 +0000 UTC m=+67.141367770" watchObservedRunningTime="2026-07-24 19:27:17.181406074 +0000 UTC m=+67.276895283" Jul 24 19:27:17.182949 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.182796 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f"] Jul 24 19:27:17.185795 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.185592 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75276\"" Jul 24 19:27:17.189418 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:17.189391 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7267d569_0982_4797_af25_32d1752be10b.slice/crio-2f06a182a9f731e3ec1ec309094309baaf2fed9de32d3f7daeaee73d60563439 WatchSource:0}: Error finding container 2f06a182a9f731e3ec1ec309094309baaf2fed9de32d3f7daeaee73d60563439: Status 404 returned error can't find the container with id 2f06a182a9f731e3ec1ec309094309baaf2fed9de32d3f7daeaee73d60563439 Jul 24 19:27:17.194716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.194446 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ghxhc" Jul 24 19:27:17.339683 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.339588 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56947db777-2bfh7"] Jul 24 19:27:17.347462 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:17.347424 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9996735c_0afe_414c_b463_b2f4c223d41c.slice/crio-3a5a01bc897c4a34ac1ccfabb3bcd4d8a77d898f6125646d1370fd3facd55805 WatchSource:0}: Error finding container 3a5a01bc897c4a34ac1ccfabb3bcd4d8a77d898f6125646d1370fd3facd55805: Status 404 returned error can't find the container with id 3a5a01bc897c4a34ac1ccfabb3bcd4d8a77d898f6125646d1370fd3facd55805 Jul 24 19:27:17.395192 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.391502 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f69ccb89-9t2qh"] Jul 24 19:27:17.398882 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:17.398821 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf808fd24_f54d_4cd6_9a39_8c3ad4258d7a.slice/crio-3545fb2d328ddf533f8cf0dea2e54654d273571dac18f1e9c8f478e616933e2c WatchSource:0}: Error finding container 3545fb2d328ddf533f8cf0dea2e54654d273571dac18f1e9c8f478e616933e2c: Status 404 returned error can't find the container with id 3545fb2d328ddf533f8cf0dea2e54654d273571dac18f1e9c8f478e616933e2c Jul 24 19:27:17.451896 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.451797 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ghxhc"] Jul 24 19:27:17.900331 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.899834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:27:17.940239 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:17.940168 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c246e701-32dc-4ec9-a53d-b4e9cffdff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-99pmx\" (UID: \"c246e701-32dc-4ec9-a53d-b4e9cffdff6b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:27:18.042870 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.042214 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" event={"ID":"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a","Type":"ContainerStarted","Data":"8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9"} Jul 24 19:27:18.042870 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.042286 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" event={"ID":"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a","Type":"ContainerStarted","Data":"3545fb2d328ddf533f8cf0dea2e54654d273571dac18f1e9c8f478e616933e2c"} Jul 24 19:27:18.042870 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.042726 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:27:18.046063 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.046002 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" event={"ID":"7267d569-0982-4797-af25-32d1752be10b","Type":"ContainerStarted","Data":"2f06a182a9f731e3ec1ec309094309baaf2fed9de32d3f7daeaee73d60563439"} Jul 24 19:27:18.065900 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.064050 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" podStartSLOduration=68.064031521 podStartE2EDuration="1m8.064031521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:18.062048913 +0000 UTC m=+68.157538123" watchObservedRunningTime="2026-07-24 19:27:18.064031521 +0000 UTC m=+68.159520726" Jul 24 19:27:18.065900 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.065136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56947db777-2bfh7" event={"ID":"9996735c-0afe-414c-b463-b2f4c223d41c","Type":"ContainerStarted","Data":"b5f20d1600807efa5fd032f92fe5941ae53bbe248dc3313a98c8dc2ee5ae4cba"} Jul 24 19:27:18.065900 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.065177 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56947db777-2bfh7" event={"ID":"9996735c-0afe-414c-b463-b2f4c223d41c","Type":"ContainerStarted","Data":"3a5a01bc897c4a34ac1ccfabb3bcd4d8a77d898f6125646d1370fd3facd55805"} Jul 24 19:27:18.069523 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.069502 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-624qv\"" Jul 24 19:27:18.081716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.081499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" Jul 24 19:27:18.085376 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.084083 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-56947db777-2bfh7" podStartSLOduration=62.084066047 podStartE2EDuration="1m2.084066047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:18.083348175 +0000 UTC m=+68.178837382" watchObservedRunningTime="2026-07-24 19:27:18.084066047 +0000 UTC m=+68.179555253" Jul 24 19:27:18.090058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.090005 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ghxhc" event={"ID":"938f432d-fa76-41c9-b82b-25f85fbb30be","Type":"ContainerStarted","Data":"abe73d42deb2ee15c076261fa39c1fae96ed0648c4f89d1d6424de2346da6201"} Jul 24 19:27:18.118966 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.118897 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:18.122358 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.122169 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:18.853912 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.852552 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d75fd7d4c-w8tgj"] Jul 24 19:27:18.857410 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.856135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:18.860648 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.859001 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 19:27:18.860648 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.859423 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lbnbp\"" Jul 24 19:27:18.864485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.861229 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 19:27:18.864485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.861446 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 19:27:18.864485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.861683 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 19:27:18.864485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.861689 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 19:27:18.864485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.862463 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-99pmx"] Jul 24 19:27:18.872838 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:18.872687 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d75fd7d4c-w8tgj"] Jul 24 19:27:19.011439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.011146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-oauth-config\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.011439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.011199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-service-ca\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.011439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.011227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-serving-cert\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.011439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.011258 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pn7\" (UniqueName: \"kubernetes.io/projected/801bd08b-f115-49df-b494-a47c9cebd491-kube-api-access-j6pn7\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.011439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.011282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-console-config\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.011439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.011304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-oauth-serving-cert\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.114828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-oauth-config\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.114882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-service-ca\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.114909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-serving-cert\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.114940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pn7\" (UniqueName: \"kubernetes.io/projected/801bd08b-f115-49df-b494-a47c9cebd491-kube-api-access-j6pn7\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.114963 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-console-config\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.114984 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-oauth-serving-cert\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.116680 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-service-ca\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.117304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.117254 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-console-config\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.118192 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.118132 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-oauth-serving-cert\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.126297 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.126256 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-serving-cert\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.130656 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.130037 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-oauth-config\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.136547 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.136170 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pn7\" (UniqueName: \"kubernetes.io/projected/801bd08b-f115-49df-b494-a47c9cebd491-kube-api-access-j6pn7\") pod \"console-6d75fd7d4c-w8tgj\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.141614 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.141530 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6l22f" event={"ID":"06dc9231-d8aa-465a-aa18-a1e9e46cbd18","Type":"ContainerStarted","Data":"094604022fb077fa67093b6367d75e697140e96bb6a06707f3c81f63f694b53c"} Jul 24 19:27:19.149348 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.148743 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" event={"ID":"c246e701-32dc-4ec9-a53d-b4e9cffdff6b","Type":"ContainerStarted","Data":"f521515669de874619e09d4222a15d7cded7c6d7309f93bf5f530e6f89cef305"} Jul 24 19:27:19.149348 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.148799 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:19.151274 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.151001 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56947db777-2bfh7" Jul 24 19:27:19.198159 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.198118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:19.464967 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.464740 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d75fd7d4c-w8tgj"] Jul 24 19:27:19.484309 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:19.484261 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801bd08b_f115_49df_b494_a47c9cebd491.slice/crio-cada60c82e7ab4c8fc948dce8362192db648ceeb391d91bae1c232ac47589c38 WatchSource:0}: Error finding container cada60c82e7ab4c8fc948dce8362192db648ceeb391d91bae1c232ac47589c38: Status 404 returned error can't find the container with id cada60c82e7ab4c8fc948dce8362192db648ceeb391d91bae1c232ac47589c38 Jul 24 19:27:19.782780 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.782726 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:19.845971 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:19.845914 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:20.173777 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:20.173684 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d75fd7d4c-w8tgj" event={"ID":"801bd08b-f115-49df-b494-a47c9cebd491","Type":"ContainerStarted","Data":"cada60c82e7ab4c8fc948dce8362192db648ceeb391d91bae1c232ac47589c38"} Jul 24 19:27:20.198752 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:20.198259 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6l22f" event={"ID":"06dc9231-d8aa-465a-aa18-a1e9e46cbd18","Type":"ContainerStarted","Data":"693dfae620961f3d3f78c80b2d832b9f6002c0991fd79e9b354f09f447a71703"} Jul 24 19:27:20.218668 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:20.216590 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6l22f" podStartSLOduration=68.041286811 podStartE2EDuration="1m10.216571004s" totalImagesPullingTime="2.175284193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:20.214683276 +0000 UTC m=+70.310172484" watchObservedRunningTime="2026-07-24 19:27:20.216571004 +0000 UTC m=+70.312060211" Jul 24 19:27:20.343601 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:20.343390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:20.420267 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:20.420219 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:21.364038 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:21.363969 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:21.422383 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:21.422309 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:22.255695 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:22.255584 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:22.357631 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:22.357107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:22.445833 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:22.445797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:22.524052 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:22.523978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:22.623990 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:22.623952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:22.688681 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:22.688637 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:22.947941 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:22.947623 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:23.010017 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:23.009899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:23.208668 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:23.208588 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:23.311930 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:23.311872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:23.619569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:23.619523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:23.748449 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:23.748398 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:27.124007 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:27.123974 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:27.198733 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:27.198693 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:27.644756 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:27.644719 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:27.713425 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:27.713395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:28.270591 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:28.270562 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:28.330329 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:28.330291 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:28.484365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:28.484288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:28.582417 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:28.582239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:30.217719 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:30.217686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:30.278872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:30.278840 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:32.270946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:32.270910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:32.326778 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:32.326748 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:34.024835 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:34.024799 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:34.081918 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:34.081843 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:37.144261 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:37.144212 2570 patch_prober.go:28] interesting pod/image-registry-5f69ccb89-9t2qh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 19:27:37.144698 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:37.144283 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 19:27:38.082343 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:38.082306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:38.153751 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:38.153713 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:39.154399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:39.154368 2570 patch_prober.go:28] interesting pod/image-registry-5f69ccb89-9t2qh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 19:27:39.154909 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:39.154418 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 19:27:40.950373 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:40.950239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:41.054690 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:41.054649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:41.114964 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:41.114932 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:42.059169 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.059138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:42.161978 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.161939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:42.242488 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.242170 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:42.299072 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.299021 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" event={"ID":"87e9f50d-5d1f-4d14-a2a4-9de20d277b1d","Type":"ContainerStarted","Data":"39d7167203dba3a9c39840a42f1661c32eda53cc6bf5ee5d2c3edfab27436f2e"} Jul 24 19:27:42.301381 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.300080 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:42.302102 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.302058 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" Jul 24 19:27:42.302221 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.302205 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h59ph" event={"ID":"d30c97b9-f517-4a45-896d-5cdf53128137","Type":"ContainerStarted","Data":"87768811f4789a36b1999afd8e0126b76a879ea619623deace98d2f94cef543a"} Jul 24 19:27:42.305327 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.305301 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" event={"ID":"7267d569-0982-4797-af25-32d1752be10b","Type":"ContainerStarted","Data":"d8d5af9616270c50f8b626825a26fbcf3ffdc519e85b37ff1b9d272fc7c5b3cc"} Jul 24 19:27:42.307651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.307617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d75fd7d4c-w8tgj" event={"ID":"801bd08b-f115-49df-b494-a47c9cebd491","Type":"ContainerStarted","Data":"b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7"} Jul 24 19:27:42.310375 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.309976 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plptd" event={"ID":"8132dea3-f97b-44d0-b452-bed78f859762","Type":"ContainerStarted","Data":"17fa7ca7f9b7277877dc9a8dbb9bd5f3ba2c0271cdca5209c66b84d44f896c58"} Jul 24 19:27:42.312692 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.312645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" event={"ID":"c246e701-32dc-4ec9-a53d-b4e9cffdff6b","Type":"ContainerStarted","Data":"873b8eaadcbe467b2ecb85e706714aa593574334fbf47bce92c926ae0650c0e0"} Jul 24 19:27:42.314682 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.314637 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" event={"ID":"c6facd7f-2d0b-4779-b131-5cab24f7337e","Type":"ContainerStarted","Data":"e55334957e02baa44dea2149b29b29b3767eafc07568d1addfa1c7ea4b08d6b6"} Jul 24 19:27:42.317875 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.317331 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" event={"ID":"618ce004-bf5d-4559-8049-ddc31cb8a3af","Type":"ContainerStarted","Data":"37d189722f9698895787f3b62ada82d249e8f42d1abd3e79ddc96ec9d9028893"} Jul 24 19:27:42.317875 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.317364 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" event={"ID":"618ce004-bf5d-4559-8049-ddc31cb8a3af","Type":"ContainerStarted","Data":"8aee9a69083efb710db8e0ce050c29a4343954c1cc853558ad26e321c56dc993"} Jul 24 19:27:42.317875 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.317381 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" event={"ID":"618ce004-bf5d-4559-8049-ddc31cb8a3af","Type":"ContainerStarted","Data":"668f144520b7ea17624549b69a68b6bf44c10be9688b4fa2b6ca3b39be0aa154"} Jul 24 19:27:42.320308 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.319917 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xrzx2" event={"ID":"d0267eb1-f96c-4d53-8651-438bb2fbcd4e","Type":"ContainerStarted","Data":"bbde4204f5c73a3cbf01b3c391aa2557ba048d89db9c396c76037cfe153b85e7"} Jul 24 19:27:42.320308 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.320276 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-xrzx2" Jul 24 19:27:42.323890 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.323856 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ghxhc" event={"ID":"938f432d-fa76-41c9-b82b-25f85fbb30be","Type":"ContainerStarted","Data":"7642455411aa3e4cf85c434411276b4c008bfff1c0104355dc18199b5ab7c15f"} Jul 24 19:27:42.324043 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.324029 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ghxhc" event={"ID":"938f432d-fa76-41c9-b82b-25f85fbb30be","Type":"ContainerStarted","Data":"13593c89d745ef2f3743014120a7fd41523c0e9d1ad2ccc31253ccc56622241b"} Jul 24 19:27:42.324408 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.324390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ghxhc" Jul 24 19:27:42.331586 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.331571 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-xrzx2" Jul 24 19:27:42.333436 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.333392 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-99pmx" podStartSLOduration=35.650822615 podStartE2EDuration="57.333374115s" totalImagesPullingTime="21.6825515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.332351155 +0000 UTC m=+92.427840362" watchObservedRunningTime="2026-07-24 19:27:42.333374115 +0000 UTC m=+92.428863323" Jul 24 19:27:42.334099 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.334065 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b67b7895b-lt9sn" podStartSLOduration=1.596518554 podStartE2EDuration="26.334055459s" totalImagesPullingTime="24.737536905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.318705571 +0000 UTC m=+92.414194778" watchObservedRunningTime="2026-07-24 19:27:42.334055459 +0000 UTC m=+92.429544668" Jul 24 19:27:42.355179 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.354747 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d75fd7d4c-w8tgj" podStartSLOduration=2.328562567 podStartE2EDuration="24.354727543s" totalImagesPullingTime="22.026164976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.353004529 +0000 UTC m=+92.448493752" watchObservedRunningTime="2026-07-24 19:27:42.354727543 +0000 UTC m=+92.450216749" Jul 24 19:27:42.371235 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.371187 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-plptd" podStartSLOduration=67.631450091 podStartE2EDuration="1m32.371173234s" totalImagesPullingTime="24.739723143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.369953923 +0000 UTC m=+92.465443130" watchObservedRunningTime="2026-07-24 19:27:42.371173234 +0000 UTC m=+92.466662440" Jul 24 19:27:42.389922 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.389870 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9f775cb7-n9s7t" podStartSLOduration=1.631120895 podStartE2EDuration="26.389838232s" totalImagesPullingTime="24.758717337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.387424243 +0000 UTC m=+92.482913451" watchObservedRunningTime="2026-07-24 19:27:42.389838232 +0000 UTC m=+92.485327440" Jul 24 19:27:42.411904 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.411861 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c96bf598b-xjm7g" podStartSLOduration=2.624355591 podStartE2EDuration="26.411842259s" totalImagesPullingTime="23.787486668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.411305181 +0000 UTC m=+92.506794388" watchObservedRunningTime="2026-07-24 19:27:42.411842259 +0000 UTC m=+92.507331469" Jul 24 19:27:42.436004 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.435949 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5xb5f" podStartSLOduration=62.093626815 podStartE2EDuration="1m26.435932445s" totalImagesPullingTime="24.34230563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.434122691 +0000 UTC m=+92.529611894" watchObservedRunningTime="2026-07-24 19:27:42.435932445 +0000 UTC m=+92.531421653" Jul 24 19:27:42.440434 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.440404 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:42.497051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.496954 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-xrzx2" podStartSLOduration=1.80192315 podStartE2EDuration="26.496934293s" totalImagesPullingTime="24.695011143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.471029106 +0000 UTC m=+92.566518313" watchObservedRunningTime="2026-07-24 19:27:42.496934293 +0000 UTC m=+92.592423503" Jul 24 19:27:42.508765 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.508543 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:42.566505 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:42.565835 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:43.265642 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:43.265548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:43.320074 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:43.320037 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:43.799039 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:43.798888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:43.853282 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:43.853164 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:44.710238 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:44.710179 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ghxhc" podStartSLOduration=36.638992221 podStartE2EDuration="1m0.710159268s" totalImagesPullingTime="24.071167047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:42.506620482 +0000 UTC m=+92.602109712" watchObservedRunningTime="2026-07-24 19:27:44.710159268 +0000 UTC m=+94.805648474" Jul 24 19:27:44.718901 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:44.714526 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f69ccb89-9t2qh"] Jul 24 19:27:44.722236 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:44.722182 2570 patch_prober.go:28] interesting pod/image-registry-5f69ccb89-9t2qh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 19:27:44.722599 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:44.722529 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 19:27:44.786575 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:44.786515 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:44.886029 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:44.885982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:44.941076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:44.940573 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:46.268536 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:46.268491 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:46.329978 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:46.329945 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:46.407618 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:46.407579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:46.497180 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:46.497146 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:46.602182 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:46.602103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:47.352616 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:47.352577 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h59ph" event={"ID":"d30c97b9-f517-4a45-896d-5cdf53128137","Type":"ContainerStarted","Data":"54743685033b824f201b96b90b899dd95a4a4d39e83535c52ab3a7b582c77753"} Jul 24 19:27:47.369399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:47.369351 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h59ph" podStartSLOduration=17.654375849 podStartE2EDuration="53.369335052s" totalImagesPullingTime="35.714959203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:26:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:47.368349927 +0000 UTC m=+97.463839133" watchObservedRunningTime="2026-07-24 19:27:47.369335052 +0000 UTC m=+97.464824258" Jul 24 19:27:47.438452 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:47.438415 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d75fd7d4c-w8tgj"] Jul 24 19:27:47.552166 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:47.552125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:47.652866 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:47.652781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:47.703382 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:47.703344 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:48.084609 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:48.084572 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:48.092941 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:48.092917 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rhks5" Jul 24 19:27:48.185026 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:48.184992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:48.721642 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:48.721602 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:49.115633 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:49.115595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:49.170485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:49.170439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:49.198550 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:49.198522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:27:49.848598 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:49.848563 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:49.945353 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:49.945324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:50.910999 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:50.910937 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:51.011152 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.011110 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:51.469833 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.469794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:51.714342 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.714292 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:51.773293 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.773260 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6c8k9"] Jul 24 19:27:51.806027 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.806000 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.812345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.812323 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 19:27:51.813998 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.813771 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 19:27:51.813998 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.813824 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tdkbv\"" Jul 24 19:27:51.814396 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.814372 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 19:27:51.817520 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.817501 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 19:27:51.854599 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.854570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:51.921146 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921103 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-tls\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921184 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmq4\" (UniqueName: \"kubernetes.io/projected/ed45484f-b97c-4364-9c25-39daa56d9f97-kube-api-access-6jmq4\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-textfile\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921247 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-root\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-wtmp\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921382 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed45484f-b97c-4364-9c25-39daa56d9f97-metrics-client-ca\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-sys\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.921569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.921456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:51.978120 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:51.978067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.021980 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.021957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-textfile\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022118 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.021996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-root\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022118 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022017 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-wtmp\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022118 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed45484f-b97c-4364-9c25-39daa56d9f97-metrics-client-ca\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-sys\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022216 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-tls\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmq4\" (UniqueName: \"kubernetes.io/projected/ed45484f-b97c-4364-9c25-39daa56d9f97-kube-api-access-6jmq4\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022242 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-wtmp\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022247 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-sys\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022711 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022304 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed45484f-b97c-4364-9c25-39daa56d9f97-root\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022711 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022327 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-textfile\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022804 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022787 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed45484f-b97c-4364-9c25-39daa56d9f97-metrics-client-ca\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.022921 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.022901 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.025025 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.024967 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.025229 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.025211 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed45484f-b97c-4364-9c25-39daa56d9f97-node-exporter-tls\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.030824 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.030784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmq4\" (UniqueName: \"kubernetes.io/projected/ed45484f-b97c-4364-9c25-39daa56d9f97-kube-api-access-6jmq4\") pod \"node-exporter-6c8k9\" (UID: \"ed45484f-b97c-4364-9c25-39daa56d9f97\") " pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.073737 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.073672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.117196 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.117170 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6c8k9" Jul 24 19:27:52.152522 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:27:52.152495 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded45484f_b97c_4364_9c25_39daa56d9f97.slice/crio-fee08ac654847f9414d1a61d6b8fc2a525eeecabc3a3accac504787604624745 WatchSource:0}: Error finding container fee08ac654847f9414d1a61d6b8fc2a525eeecabc3a3accac504787604624745: Status 404 returned error can't find the container with id fee08ac654847f9414d1a61d6b8fc2a525eeecabc3a3accac504787604624745 Jul 24 19:27:52.189589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.189564 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.288013 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.287945 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.335069 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.335044 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ghxhc" Jul 24 19:27:52.374951 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.374916 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6c8k9" event={"ID":"ed45484f-b97c-4364-9c25-39daa56d9f97","Type":"ContainerStarted","Data":"fee08ac654847f9414d1a61d6b8fc2a525eeecabc3a3accac504787604624745"} Jul 24 19:27:52.416604 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.416578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.517095 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.517050 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.611762 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.611677 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.713135 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.713069 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:52.816996 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:52.816941 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:54.384126 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:54.384095 2570 generic.go:356] "Generic (PLEG): container finished" podID="ed45484f-b97c-4364-9c25-39daa56d9f97" containerID="9f90db8292ca158be9698a3ee346ce2e0cfd40b24b02e77733b10e166bf6f374" exitCode=0 Jul 24 19:27:54.384592 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:54.384186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6c8k9" event={"ID":"ed45484f-b97c-4364-9c25-39daa56d9f97","Type":"ContainerDied","Data":"9f90db8292ca158be9698a3ee346ce2e0cfd40b24b02e77733b10e166bf6f374"} Jul 24 19:27:54.719273 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:54.719199 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:27:54.796158 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:54.796120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:54.884236 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:54.884208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:54.981678 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:54.981566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:55.390417 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:55.390379 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6c8k9" event={"ID":"ed45484f-b97c-4364-9c25-39daa56d9f97","Type":"ContainerStarted","Data":"60ae465b995ad8481797d7f00a80e6adaaaa43c407d026c56e6e1d94d606cc03"} Jul 24 19:27:55.390895 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:55.390447 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6c8k9" event={"ID":"ed45484f-b97c-4364-9c25-39daa56d9f97","Type":"ContainerStarted","Data":"6a39615f7b1ac39d453d947a8e72943cd51ba7a5d9b5e566a389ec50d1e18621"} Jul 24 19:27:55.420846 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:55.420801 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6c8k9" podStartSLOduration=2.8670171570000003 podStartE2EDuration="4.420786502s" totalImagesPullingTime="1.46808747s" totalInitContainerRuntime="85.681875ms" isStatefulPod=true podCreationTimestamp="2026-07-24 19:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:55.42030252 +0000 UTC m=+105.515791726" watchObservedRunningTime="2026-07-24 19:27:55.420786502 +0000 UTC m=+105.516275729" Jul 24 19:27:55.491394 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:55.491372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:55.589303 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:55.589270 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:56.491183 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:56.491148 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:56.589820 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:56.589775 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:56.693628 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:56.693601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:56.791829 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:56.791746 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:56.889614 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:56.889580 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:58.013692 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:58.013651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:27:58.861266 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:27:58.861232 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:00.413932 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:00.413899 2570 generic.go:356] "Generic (PLEG): container finished" podID="e2a9c64f-a5aa-4d52-8981-9aaceb773a22" containerID="3acef5cf490488113bdb3d4f6ed639317d6bb890c84248a8f8ab8ec36fcbfd15" exitCode=0 Jul 24 19:28:00.414315 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:00.413972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" event={"ID":"e2a9c64f-a5aa-4d52-8981-9aaceb773a22","Type":"ContainerDied","Data":"3acef5cf490488113bdb3d4f6ed639317d6bb890c84248a8f8ab8ec36fcbfd15"} Jul 24 19:28:00.414315 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:00.414268 2570 scope.go:117] "RemoveContainer" containerID="3acef5cf490488113bdb3d4f6ed639317d6bb890c84248a8f8ab8ec36fcbfd15" Jul 24 19:28:00.671012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:00.670923 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:00.770710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:00.770674 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:01.419754 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:01.419719 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qblst" event={"ID":"e2a9c64f-a5aa-4d52-8981-9aaceb773a22","Type":"ContainerStarted","Data":"7348bf31e4f8ed2b88c67e65b888093b27a908acd4a78d1ddff2ffc15e1c8f9d"} Jul 24 19:28:07.837262 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:07.837213 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:07.936661 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:07.936628 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:09.449880 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:09.449848 2570 generic.go:356] "Generic (PLEG): container finished" podID="faeb92d0-d3b8-4602-95ce-941edc7337dd" containerID="2c28d8b5b84d11edb91368da59403e80c2e67fb378175ed7c8dde22fba5ad252" exitCode=0 Jul 24 19:28:09.450203 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:09.449894 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-s6q2j" event={"ID":"faeb92d0-d3b8-4602-95ce-941edc7337dd","Type":"ContainerDied","Data":"2c28d8b5b84d11edb91368da59403e80c2e67fb378175ed7c8dde22fba5ad252"} Jul 24 19:28:09.450203 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:09.450192 2570 scope.go:117] "RemoveContainer" containerID="2c28d8b5b84d11edb91368da59403e80c2e67fb378175ed7c8dde22fba5ad252" Jul 24 19:28:09.743069 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:09.743008 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerName="registry" containerID="cri-o://8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9" gracePeriod=30 Jul 24 19:28:10.040710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.040686 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172252 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172315 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-bound-sa-token\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172344 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-certificates\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172381 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-installation-pull-secrets\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172433 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-ca-trust-extracted\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172481 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-image-registry-private-configuration\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172518 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-kube-api-access-ltf6j\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173012 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.172586 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-trusted-ca\") pod \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\" (UID: \"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a\") " Jul 24 19:28:10.173498 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.173398 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:28:10.173782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.173741 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:28:10.176298 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.176128 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-kube-api-access-ltf6j" (OuterVolumeSpecName: "kube-api-access-ltf6j") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "kube-api-access-ltf6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:10.176298 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.176274 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:10.176574 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.176536 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 19:28:10.177029 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.176995 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 19:28:10.177348 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.177316 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:10.188304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.188269 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" (UID: "f808fd24-f54d-4cd6-9a39-8c3ad4258d7a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:28:10.273995 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.273922 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-trusted-ca\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.273995 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.273957 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-tls\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.273995 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.273971 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-bound-sa-token\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.273995 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.273980 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-registry-certificates\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.273995 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.273990 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-installation-pull-secrets\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.273995 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.273998 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-ca-trust-extracted\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.274263 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.274008 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-image-registry-private-configuration\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.274263 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.274016 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a-kube-api-access-ltf6j\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.453860 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.453831 2570 generic.go:356] "Generic (PLEG): container finished" podID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerID="8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9" exitCode=0 Jul 24 19:28:10.454227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.453872 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" Jul 24 19:28:10.454227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.453918 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" event={"ID":"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a","Type":"ContainerDied","Data":"8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9"} Jul 24 19:28:10.454227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.453978 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f69ccb89-9t2qh" event={"ID":"f808fd24-f54d-4cd6-9a39-8c3ad4258d7a","Type":"ContainerDied","Data":"3545fb2d328ddf533f8cf0dea2e54654d273571dac18f1e9c8f478e616933e2c"} Jul 24 19:28:10.454227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.454011 2570 scope.go:117] "RemoveContainer" containerID="8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9" Jul 24 19:28:10.455822 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.455796 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-s6q2j" event={"ID":"faeb92d0-d3b8-4602-95ce-941edc7337dd","Type":"ContainerStarted","Data":"1ed2d022bfe0b27d20120cfacc4fd93297605b9428e800c6fa0e963ee41c664e"} Jul 24 19:28:10.462795 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.462780 2570 scope.go:117] "RemoveContainer" containerID="8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9" Jul 24 19:28:10.463048 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:28:10.463024 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9\": container with ID starting with 8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9 not found: ID does not exist" containerID="8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9" Jul 24 19:28:10.463112 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.463060 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9"} err="failed to get container status \"8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9\": rpc error: code = NotFound desc = could not find container \"8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9\": container with ID starting with 8162ad8a94696a2aafb16ed2115247d5a85bf9c3e47d60a881d545811102dbe9 not found: ID does not exist" Jul 24 19:28:10.487329 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.487306 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f69ccb89-9t2qh"] Jul 24 19:28:10.492284 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.492265 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f69ccb89-9t2qh"] Jul 24 19:28:10.584826 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:10.584759 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" path="/var/lib/kubelet/pods/f808fd24-f54d-4cd6-9a39-8c3ad4258d7a/volumes" Jul 24 19:28:12.461352 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.461288 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d75fd7d4c-w8tgj" podUID="801bd08b-f115-49df-b494-a47c9cebd491" containerName="console" containerID="cri-o://b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7" gracePeriod=15 Jul 24 19:28:12.723345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.723281 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:28:12.755519 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.755487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:12.797371 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797337 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-oauth-config\") pod \"801bd08b-f115-49df-b494-a47c9cebd491\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " Jul 24 19:28:12.797533 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797385 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-console-config\") pod \"801bd08b-f115-49df-b494-a47c9cebd491\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " Jul 24 19:28:12.797533 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797406 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-oauth-serving-cert\") pod \"801bd08b-f115-49df-b494-a47c9cebd491\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " Jul 24 19:28:12.797533 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797428 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-serving-cert\") pod \"801bd08b-f115-49df-b494-a47c9cebd491\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " Jul 24 19:28:12.797533 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797504 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6pn7\" (UniqueName: \"kubernetes.io/projected/801bd08b-f115-49df-b494-a47c9cebd491-kube-api-access-j6pn7\") pod \"801bd08b-f115-49df-b494-a47c9cebd491\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " Jul 24 19:28:12.797762 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797578 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-service-ca\") pod \"801bd08b-f115-49df-b494-a47c9cebd491\" (UID: \"801bd08b-f115-49df-b494-a47c9cebd491\") " Jul 24 19:28:12.797891 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797863 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-console-config" (OuterVolumeSpecName: "console-config") pod "801bd08b-f115-49df-b494-a47c9cebd491" (UID: "801bd08b-f115-49df-b494-a47c9cebd491"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:28:12.797977 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.797868 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "801bd08b-f115-49df-b494-a47c9cebd491" (UID: "801bd08b-f115-49df-b494-a47c9cebd491"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:28:12.798111 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.798068 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-service-ca" (OuterVolumeSpecName: "service-ca") pod "801bd08b-f115-49df-b494-a47c9cebd491" (UID: "801bd08b-f115-49df-b494-a47c9cebd491"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:28:12.799941 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.799916 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801bd08b-f115-49df-b494-a47c9cebd491-kube-api-access-j6pn7" (OuterVolumeSpecName: "kube-api-access-j6pn7") pod "801bd08b-f115-49df-b494-a47c9cebd491" (UID: "801bd08b-f115-49df-b494-a47c9cebd491"). InnerVolumeSpecName "kube-api-access-j6pn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:12.800026 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.799941 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "801bd08b-f115-49df-b494-a47c9cebd491" (UID: "801bd08b-f115-49df-b494-a47c9cebd491"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 19:28:12.800026 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.799989 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "801bd08b-f115-49df-b494-a47c9cebd491" (UID: "801bd08b-f115-49df-b494-a47c9cebd491"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 19:28:12.812677 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.812655 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:12.898968 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.898934 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6pn7\" (UniqueName: \"kubernetes.io/projected/801bd08b-f115-49df-b494-a47c9cebd491-kube-api-access-j6pn7\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:12.898968 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.898965 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-service-ca\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:12.898968 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.898976 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-oauth-config\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:12.899217 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.898984 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-console-config\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:12.899217 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.898993 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/801bd08b-f115-49df-b494-a47c9cebd491-oauth-serving-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:12.899217 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.899005 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/801bd08b-f115-49df-b494-a47c9cebd491-console-serving-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:28:12.913681 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:12.913648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:13.267677 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.267643 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:13.363384 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.363344 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:28:13.470094 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.470057 2570 generic.go:356] "Generic (PLEG): container finished" podID="801bd08b-f115-49df-b494-a47c9cebd491" containerID="b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7" exitCode=0 Jul 24 19:28:13.470546 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.470107 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d75fd7d4c-w8tgj" Jul 24 19:28:13.470546 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.470145 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d75fd7d4c-w8tgj" event={"ID":"801bd08b-f115-49df-b494-a47c9cebd491","Type":"ContainerDied","Data":"b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7"} Jul 24 19:28:13.470546 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.470197 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d75fd7d4c-w8tgj" event={"ID":"801bd08b-f115-49df-b494-a47c9cebd491","Type":"ContainerDied","Data":"cada60c82e7ab4c8fc948dce8362192db648ceeb391d91bae1c232ac47589c38"} Jul 24 19:28:13.470546 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.470219 2570 scope.go:117] "RemoveContainer" containerID="b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7" Jul 24 19:28:13.478893 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.478872 2570 scope.go:117] "RemoveContainer" containerID="b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7" Jul 24 19:28:13.479139 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:28:13.479119 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7\": container with ID starting with b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7 not found: ID does not exist" containerID="b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7" Jul 24 19:28:13.479232 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.479144 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7"} err="failed to get container status \"b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7\": rpc error: code = NotFound desc = could not find container \"b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7\": container with ID starting with b6e8c2e320aa6982225feaf6d25e570d0e375c975adfd704870494d5dd717ec7 not found: ID does not exist" Jul 24 19:28:13.490840 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.490818 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d75fd7d4c-w8tgj"] Jul 24 19:28:13.496392 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:13.496372 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d75fd7d4c-w8tgj"] Jul 24 19:28:14.579453 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:14.579424 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801bd08b-f115-49df-b494-a47c9cebd491" path="/var/lib/kubelet/pods/801bd08b-f115-49df-b494-a47c9cebd491/volumes" Jul 24 19:28:15.480017 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:15.479985 2570 generic.go:356] "Generic (PLEG): container finished" podID="fc62b1cf-7e3a-4df4-9272-82d6e407dfec" containerID="60be0c9f3d230f1c68d3e85e6f0a2c46553b0abbddf0c5cf8eaab7ce59108352" exitCode=0 Jul 24 19:28:15.480183 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:15.480057 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" event={"ID":"fc62b1cf-7e3a-4df4-9272-82d6e407dfec","Type":"ContainerDied","Data":"60be0c9f3d230f1c68d3e85e6f0a2c46553b0abbddf0c5cf8eaab7ce59108352"} Jul 24 19:28:15.480408 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:15.480394 2570 scope.go:117] "RemoveContainer" containerID="60be0c9f3d230f1c68d3e85e6f0a2c46553b0abbddf0c5cf8eaab7ce59108352" Jul 24 19:28:16.485752 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:16.485716 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2" event={"ID":"fc62b1cf-7e3a-4df4-9272-82d6e407dfec","Type":"ContainerStarted","Data":"bd06245d89e0ec63fc1d732b425c4277416a66d77a2bb83b834a60e3fb6c4fd9"} Jul 24 19:28:30.845044 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:28:30.844991 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:07.927064 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:07.927026 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:07.980340 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:07.980307 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:11.433970 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:11.433931 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:19.848190 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:19.848140 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:19.905414 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:19.905388 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:22.019452 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:22.019413 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:22.119511 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:22.119455 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:22.180961 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:22.180927 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:28.282913 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:28.282872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:28.383829 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:28.383797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:40.461330 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:40.461294 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:40.514960 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:40.514924 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:49.297298 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:49.297263 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:51.797952 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:51.797918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:51.900087 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:51.900055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:29:56.858726 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:29:56.858690 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:13.799051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:13.799016 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:15.323630 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:15.323596 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:17.608984 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:17.608945 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:17.670888 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:17.670858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:19.480782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:19.480720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:19.636430 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:19.636396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:19.848716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:19.848684 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:19.982696 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:19.982649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:21.680367 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:21.680302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:22.086881 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:22.086848 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:24.198367 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:24.198336 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:24.267652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:24.267601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:27.459961 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:27.459921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:27.561656 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:27.561621 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:30.066879 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:30.066843 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:34.611933 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:34.611851 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:34.712066 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:34.712026 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:41.179899 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:41.179857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:41.290867 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:41.290832 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:43.369043 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:43.368988 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:47.589243 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:47.589212 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:47.671958 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:47.671928 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:54.997714 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:54.997660 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:55.097942 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:55.097912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:55.663412 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:55.663378 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:55.914810 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:55.914705 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:56.013388 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.013337 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:56.040008 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.039980 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk"] Jul 24 19:30:56.040255 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.040244 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801bd08b-f115-49df-b494-a47c9cebd491" containerName="console" Jul 24 19:30:56.040300 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.040256 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="801bd08b-f115-49df-b494-a47c9cebd491" containerName="console" Jul 24 19:30:56.040300 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.040269 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerName="registry" Jul 24 19:30:56.040300 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.040276 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerName="registry" Jul 24 19:30:56.040388 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.040329 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="801bd08b-f115-49df-b494-a47c9cebd491" containerName="console" Jul 24 19:30:56.040388 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.040340 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f808fd24-f54d-4cd6-9a39-8c3ad4258d7a" containerName="registry" Jul 24 19:30:56.044531 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.044508 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.050499 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.050461 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 19:30:56.050759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.050743 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 19:30:56.050852 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.050793 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 19:30:56.050852 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.050840 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k4bkj\"" Jul 24 19:30:56.051056 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.051036 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 19:30:56.066484 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.066447 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk"] Jul 24 19:30:56.082091 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.082069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b89458d2-31af-4662-93db-0378273ff84f-tmp\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.082210 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.082131 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b89458d2-31af-4662-93db-0378273ff84f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.082210 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.082155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72p9\" (UniqueName: \"kubernetes.io/projected/b89458d2-31af-4662-93db-0378273ff84f-kube-api-access-v72p9\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.082278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.082232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b89458d2-31af-4662-93db-0378273ff84f-manifests\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.082278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.082268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b89458d2-31af-4662-93db-0378273ff84f-webhook-cert\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.182833 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.182756 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b89458d2-31af-4662-93db-0378273ff84f-tmp\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.182833 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.182812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b89458d2-31af-4662-93db-0378273ff84f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.183001 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.182831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72p9\" (UniqueName: \"kubernetes.io/projected/b89458d2-31af-4662-93db-0378273ff84f-kube-api-access-v72p9\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.183001 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.182871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b89458d2-31af-4662-93db-0378273ff84f-manifests\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.183001 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.182906 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b89458d2-31af-4662-93db-0378273ff84f-webhook-cert\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.183244 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.183225 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b89458d2-31af-4662-93db-0378273ff84f-manifests\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.185087 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.185062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b89458d2-31af-4662-93db-0378273ff84f-tmp\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.185291 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.185275 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b89458d2-31af-4662-93db-0378273ff84f-webhook-cert\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.185345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.185330 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b89458d2-31af-4662-93db-0378273ff84f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.198255 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.198236 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72p9\" (UniqueName: \"kubernetes.io/projected/b89458d2-31af-4662-93db-0378273ff84f-kube-api-access-v72p9\") pod \"opendatahub-operator-controller-manager-69f558d7f4-dr6wk\" (UID: \"b89458d2-31af-4662-93db-0378273ff84f\") " pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.231290 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.231264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:56.329316 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.329284 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:56.356169 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.356153 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:30:56.430124 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.430079 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:56.486516 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.486491 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk"] Jul 24 19:30:56.489247 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:30:56.489219 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89458d2_31af_4662_93db_0378273ff84f.slice/crio-e37ecdf63fed18f1cdce2590f4690e51c862193fbbccf4ca9bef38f4ab59aa23 WatchSource:0}: Error finding container e37ecdf63fed18f1cdce2590f4690e51c862193fbbccf4ca9bef38f4ab59aa23: Status 404 returned error can't find the container with id e37ecdf63fed18f1cdce2590f4690e51c862193fbbccf4ca9bef38f4ab59aa23 Jul 24 19:30:56.519627 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.519600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:56.590369 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:56.590337 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:30:57.103457 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:30:57.103422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" event={"ID":"b89458d2-31af-4662-93db-0378273ff84f","Type":"ContainerStarted","Data":"e37ecdf63fed18f1cdce2590f4690e51c862193fbbccf4ca9bef38f4ab59aa23"} Jul 24 19:31:00.118921 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:00.118886 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" event={"ID":"b89458d2-31af-4662-93db-0378273ff84f","Type":"ContainerStarted","Data":"2e0c022d61d860afbdf4aa90b4b0e2b0f181eb4de3869dfd1254d43fb4fa3fb4"} Jul 24 19:31:01.124962 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:01.124918 2570 generic.go:356] "Generic (PLEG): container finished" podID="b89458d2-31af-4662-93db-0378273ff84f" containerID="2e0c022d61d860afbdf4aa90b4b0e2b0f181eb4de3869dfd1254d43fb4fa3fb4" exitCode=0 Jul 24 19:31:01.125395 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:01.125002 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" event={"ID":"b89458d2-31af-4662-93db-0378273ff84f","Type":"ContainerDied","Data":"2e0c022d61d860afbdf4aa90b4b0e2b0f181eb4de3869dfd1254d43fb4fa3fb4"} Jul 24 19:31:02.131781 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:02.131739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" event={"ID":"b89458d2-31af-4662-93db-0378273ff84f","Type":"ContainerStarted","Data":"65eb7151200f052a332d3f3943e117f9b39cf2acb7c2a41e6a07f9821257a731"} Jul 24 19:31:02.132278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:02.131904 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:31:02.163378 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:02.163327 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" podStartSLOduration=1.6179044 podStartE2EDuration="6.163306809s" totalImagesPullingTime="3.473584293s" totalInitContainerRuntime="1.071818116s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:30:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:02.162018076 +0000 UTC m=+292.257507283" watchObservedRunningTime="2026-07-24 19:31:02.163306809 +0000 UTC m=+292.258796016" Jul 24 19:31:10.317120 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:10.317086 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:31:10.324183 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:10.324154 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:31:10.329574 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:10.329549 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 19:31:11.902291 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:11.902249 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:11.962027 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:11.961996 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:13.138719 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:13.138691 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69f558d7f4-dr6wk" Jul 24 19:31:15.221519 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:15.221487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:15.979482 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:15.979422 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j7z9v"] Jul 24 19:31:15.987973 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:15.987946 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:15.990886 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:15.990857 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 19:31:15.991322 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:15.991306 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qz924\"" Jul 24 19:31:16.011041 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.011012 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j7z9v"] Jul 24 19:31:16.046252 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.046216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab251975-fd07-4a97-9093-a4d55deca6c0-cert\") pod \"odh-model-controller-7998c67dc4-j7z9v\" (UID: \"ab251975-fd07-4a97-9093-a4d55deca6c0\") " pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:16.046438 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.046326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9h7\" (UniqueName: \"kubernetes.io/projected/ab251975-fd07-4a97-9093-a4d55deca6c0-kube-api-access-jr9h7\") pod \"odh-model-controller-7998c67dc4-j7z9v\" (UID: \"ab251975-fd07-4a97-9093-a4d55deca6c0\") " pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:16.147621 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.147586 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab251975-fd07-4a97-9093-a4d55deca6c0-cert\") pod \"odh-model-controller-7998c67dc4-j7z9v\" (UID: \"ab251975-fd07-4a97-9093-a4d55deca6c0\") " pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:16.147836 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.147653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9h7\" (UniqueName: \"kubernetes.io/projected/ab251975-fd07-4a97-9093-a4d55deca6c0-kube-api-access-jr9h7\") pod \"odh-model-controller-7998c67dc4-j7z9v\" (UID: \"ab251975-fd07-4a97-9093-a4d55deca6c0\") " pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:16.147836 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:16.147773 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 19:31:16.147961 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:16.147862 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab251975-fd07-4a97-9093-a4d55deca6c0-cert podName:ab251975-fd07-4a97-9093-a4d55deca6c0 nodeName:}" failed. No retries permitted until 2026-07-24 19:31:16.64783741 +0000 UTC m=+306.743326595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ab251975-fd07-4a97-9093-a4d55deca6c0-cert") pod "odh-model-controller-7998c67dc4-j7z9v" (UID: "ab251975-fd07-4a97-9093-a4d55deca6c0") : secret "odh-model-controller-webhook-cert" not found Jul 24 19:31:16.158267 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.158226 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9h7\" (UniqueName: \"kubernetes.io/projected/ab251975-fd07-4a97-9093-a4d55deca6c0-kube-api-access-jr9h7\") pod \"odh-model-controller-7998c67dc4-j7z9v\" (UID: \"ab251975-fd07-4a97-9093-a4d55deca6c0\") " pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:16.376824 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.376789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:16.609618 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.609562 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:16.617999 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.617966 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r82sc"] Jul 24 19:31:16.623617 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.623593 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:16.626296 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.626276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 19:31:16.626717 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.626700 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vg28k\"" Jul 24 19:31:16.631811 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.631750 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r82sc"] Jul 24 19:31:16.652645 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.652622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab251975-fd07-4a97-9093-a4d55deca6c0-cert\") pod \"odh-model-controller-7998c67dc4-j7z9v\" (UID: \"ab251975-fd07-4a97-9093-a4d55deca6c0\") " pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:16.655197 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.655167 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab251975-fd07-4a97-9093-a4d55deca6c0-cert\") pod \"odh-model-controller-7998c67dc4-j7z9v\" (UID: \"ab251975-fd07-4a97-9093-a4d55deca6c0\") " pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:16.734944 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.734901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:16.753759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.753721 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf26g\" (UniqueName: \"kubernetes.io/projected/1cd435c2-ab14-43f0-8728-1f64a2b193b3-kube-api-access-jf26g\") pod \"model-serving-api-54d89bd5fc-r82sc\" (UID: \"1cd435c2-ab14-43f0-8728-1f64a2b193b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:16.753923 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.753767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd435c2-ab14-43f0-8728-1f64a2b193b3-tls-certs\") pod \"model-serving-api-54d89bd5fc-r82sc\" (UID: \"1cd435c2-ab14-43f0-8728-1f64a2b193b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:16.838772 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.838735 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:16.854973 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.854945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd435c2-ab14-43f0-8728-1f64a2b193b3-tls-certs\") pod \"model-serving-api-54d89bd5fc-r82sc\" (UID: \"1cd435c2-ab14-43f0-8728-1f64a2b193b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:16.855139 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.855019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf26g\" (UniqueName: \"kubernetes.io/projected/1cd435c2-ab14-43f0-8728-1f64a2b193b3-kube-api-access-jf26g\") pod \"model-serving-api-54d89bd5fc-r82sc\" (UID: \"1cd435c2-ab14-43f0-8728-1f64a2b193b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:16.855139 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:16.855094 2570 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 24 19:31:16.855255 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:16.855166 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd435c2-ab14-43f0-8728-1f64a2b193b3-tls-certs podName:1cd435c2-ab14-43f0-8728-1f64a2b193b3 nodeName:}" failed. No retries permitted until 2026-07-24 19:31:17.355149437 +0000 UTC m=+307.450638621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1cd435c2-ab14-43f0-8728-1f64a2b193b3-tls-certs") pod "model-serving-api-54d89bd5fc-r82sc" (UID: "1cd435c2-ab14-43f0-8728-1f64a2b193b3") : secret "model-serving-api-tls" not found Jul 24 19:31:16.871282 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.871256 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf26g\" (UniqueName: \"kubernetes.io/projected/1cd435c2-ab14-43f0-8728-1f64a2b193b3-kube-api-access-jf26g\") pod \"model-serving-api-54d89bd5fc-r82sc\" (UID: \"1cd435c2-ab14-43f0-8728-1f64a2b193b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:16.900153 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:16.900082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:17.012053 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.012016 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:17.045338 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.045250 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j7z9v"] Jul 24 19:31:17.048171 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:31:17.048142 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab251975_fd07_4a97_9093_a4d55deca6c0.slice/crio-dec29dc87a59cb734d80e3ef186873637c59ab8ca3ad8786a26b91a480b97243 WatchSource:0}: Error finding container dec29dc87a59cb734d80e3ef186873637c59ab8ca3ad8786a26b91a480b97243: Status 404 returned error can't find the container with id dec29dc87a59cb734d80e3ef186873637c59ab8ca3ad8786a26b91a480b97243 Jul 24 19:31:17.049542 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.049525 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 19:31:17.197390 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.197299 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" event={"ID":"ab251975-fd07-4a97-9093-a4d55deca6c0","Type":"ContainerStarted","Data":"dec29dc87a59cb734d80e3ef186873637c59ab8ca3ad8786a26b91a480b97243"} Jul 24 19:31:17.358694 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.358648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd435c2-ab14-43f0-8728-1f64a2b193b3-tls-certs\") pod \"model-serving-api-54d89bd5fc-r82sc\" (UID: \"1cd435c2-ab14-43f0-8728-1f64a2b193b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:17.361239 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.361215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cd435c2-ab14-43f0-8728-1f64a2b193b3-tls-certs\") pod \"model-serving-api-54d89bd5fc-r82sc\" (UID: \"1cd435c2-ab14-43f0-8728-1f64a2b193b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:17.536520 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.536492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:17.675569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:17.675504 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r82sc"] Jul 24 19:31:17.678821 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:31:17.678787 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd435c2_ab14_43f0_8728_1f64a2b193b3.slice/crio-5b935c9c2f04c827d5814242d79ee5292cd705f4eec1f75415416740483c3178 WatchSource:0}: Error finding container 5b935c9c2f04c827d5814242d79ee5292cd705f4eec1f75415416740483c3178: Status 404 returned error can't find the container with id 5b935c9c2f04c827d5814242d79ee5292cd705f4eec1f75415416740483c3178 Jul 24 19:31:18.205262 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:18.205219 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" event={"ID":"1cd435c2-ab14-43f0-8728-1f64a2b193b3","Type":"ContainerStarted","Data":"5b935c9c2f04c827d5814242d79ee5292cd705f4eec1f75415416740483c3178"} Jul 24 19:31:21.221215 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:21.221178 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" event={"ID":"ab251975-fd07-4a97-9093-a4d55deca6c0","Type":"ContainerStarted","Data":"cddbac1aaf39f60d8ee3db29ff001013242bbe3077e14430a7c9d2a5b6f3e3e2"} Jul 24 19:31:21.221703 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:21.221332 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:21.222721 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:21.222697 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" event={"ID":"1cd435c2-ab14-43f0-8728-1f64a2b193b3","Type":"ContainerStarted","Data":"db2dc26b5869d870d048ed9d79e5b9d26d064faf2bde251305613bf21e9cbf6b"} Jul 24 19:31:21.222845 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:21.222807 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:21.238624 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:21.238572 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" podStartSLOduration=2.823925805 podStartE2EDuration="6.238558884s" totalImagesPullingTime="3.414633079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:21.237303567 +0000 UTC m=+311.332792775" watchObservedRunningTime="2026-07-24 19:31:21.238558884 +0000 UTC m=+311.334048090" Jul 24 19:31:21.254293 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:21.254252 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" podStartSLOduration=2.473685992 podStartE2EDuration="5.254238875s" totalImagesPullingTime="2.780552883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:21.252714873 +0000 UTC m=+311.348204081" watchObservedRunningTime="2026-07-24 19:31:21.254238875 +0000 UTC m=+311.349728103" Jul 24 19:31:26.282825 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.282791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:26.385140 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.385097 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:26.394846 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.394814 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h"] Jul 24 19:31:26.398326 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.398304 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.402304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.402236 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 19:31:26.402445 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.402421 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 19:31:26.417141 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.417119 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h"] Jul 24 19:31:26.430483 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.430443 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2"] Jul 24 19:31:26.434939 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.434921 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.437119 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.437087 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch"] Jul 24 19:31:26.438133 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.438115 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 19:31:26.438334 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.438319 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9292s\"" Jul 24 19:31:26.440966 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.440941 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:26.443782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.443355 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 19:31:26.443782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.443406 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pgmn2\"" Jul 24 19:31:26.460322 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.459784 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2"] Jul 24 19:31:26.463225 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.463122 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch"] Jul 24 19:31:26.544144 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.544061 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6tch\" (UID: \"7ef36049-ec6f-4beb-9e2f-fbee5edb3920\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:26.544144 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.544109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tls-certs\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.544380 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.544159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpdj\" (UniqueName: \"kubernetes.io/projected/be6fb9cb-80af-4da4-be5f-a7a4089505f8-kube-api-access-zzpdj\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.544380 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.544185 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54717a5e-11ed-46fb-98a4-a7613ba540d5-cert\") pod \"kserve-controller-manager-d6f4b65b8-vgqt2\" (UID: \"54717a5e-11ed-46fb-98a4-a7613ba540d5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.544380 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.544278 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tmp\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.544380 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.544323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk97v\" (UniqueName: \"kubernetes.io/projected/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-kube-api-access-sk97v\") pod \"llmisvc-controller-manager-777c6bd7d5-t6tch\" (UID: \"7ef36049-ec6f-4beb-9e2f-fbee5edb3920\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:26.544578 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.544378 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwh7\" (UniqueName: \"kubernetes.io/projected/54717a5e-11ed-46fb-98a4-a7613ba540d5-kube-api-access-zvwh7\") pod \"kserve-controller-manager-d6f4b65b8-vgqt2\" (UID: \"54717a5e-11ed-46fb-98a4-a7613ba540d5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.582224 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.582195 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:26.645460 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.645428 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpdj\" (UniqueName: \"kubernetes.io/projected/be6fb9cb-80af-4da4-be5f-a7a4089505f8-kube-api-access-zzpdj\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.645616 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.645498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54717a5e-11ed-46fb-98a4-a7613ba540d5-cert\") pod \"kserve-controller-manager-d6f4b65b8-vgqt2\" (UID: \"54717a5e-11ed-46fb-98a4-a7613ba540d5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.645616 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.645561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tmp\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.645616 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.645591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk97v\" (UniqueName: \"kubernetes.io/projected/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-kube-api-access-sk97v\") pod \"llmisvc-controller-manager-777c6bd7d5-t6tch\" (UID: \"7ef36049-ec6f-4beb-9e2f-fbee5edb3920\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:26.645872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.645637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwh7\" (UniqueName: \"kubernetes.io/projected/54717a5e-11ed-46fb-98a4-a7613ba540d5-kube-api-access-zvwh7\") pod \"kserve-controller-manager-d6f4b65b8-vgqt2\" (UID: \"54717a5e-11ed-46fb-98a4-a7613ba540d5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.645872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.645671 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6tch\" (UID: \"7ef36049-ec6f-4beb-9e2f-fbee5edb3920\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:26.645872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.645694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tls-certs\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.645872 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:26.645773 2570 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 24 19:31:26.645872 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:26.645797 2570 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 24 19:31:26.645872 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:26.645848 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tls-certs podName:be6fb9cb-80af-4da4-be5f-a7a4089505f8 nodeName:}" failed. No retries permitted until 2026-07-24 19:31:27.145828895 +0000 UTC m=+317.241318091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tls-certs") pod "kube-auth-proxy-cff79d5b6-t5f5h" (UID: "be6fb9cb-80af-4da4-be5f-a7a4089505f8") : secret "kube-auth-proxy-tls" not found Jul 24 19:31:26.645872 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:31:26.645863 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-cert podName:7ef36049-ec6f-4beb-9e2f-fbee5edb3920 nodeName:}" failed. No retries permitted until 2026-07-24 19:31:27.14585444 +0000 UTC m=+317.241343640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-cert") pod "llmisvc-controller-manager-777c6bd7d5-t6tch" (UID: "7ef36049-ec6f-4beb-9e2f-fbee5edb3920") : secret "llmisvc-webhook-server-cert" not found Jul 24 19:31:26.648105 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.648079 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tmp\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.648205 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.648157 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54717a5e-11ed-46fb-98a4-a7613ba540d5-cert\") pod \"kserve-controller-manager-d6f4b65b8-vgqt2\" (UID: \"54717a5e-11ed-46fb-98a4-a7613ba540d5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.667133 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.667104 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk97v\" (UniqueName: \"kubernetes.io/projected/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-kube-api-access-sk97v\") pod \"llmisvc-controller-manager-777c6bd7d5-t6tch\" (UID: \"7ef36049-ec6f-4beb-9e2f-fbee5edb3920\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:26.684799 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.684771 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpdj\" (UniqueName: \"kubernetes.io/projected/be6fb9cb-80af-4da4-be5f-a7a4089505f8-kube-api-access-zzpdj\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:26.686859 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.686830 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwh7\" (UniqueName: \"kubernetes.io/projected/54717a5e-11ed-46fb-98a4-a7613ba540d5-kube-api-access-zvwh7\") pod \"kserve-controller-manager-d6f4b65b8-vgqt2\" (UID: \"54717a5e-11ed-46fb-98a4-a7613ba540d5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.748585 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.748563 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:26.823041 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.822973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:26.906964 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.906940 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2"] Jul 24 19:31:26.908902 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:31:26.908874 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54717a5e_11ed_46fb_98a4_a7613ba540d5.slice/crio-b1a1243ced70e35a979f32874aac818c07ea04b80427495a7b119fda71592572 WatchSource:0}: Error finding container b1a1243ced70e35a979f32874aac818c07ea04b80427495a7b119fda71592572: Status 404 returned error can't find the container with id b1a1243ced70e35a979f32874aac818c07ea04b80427495a7b119fda71592572 Jul 24 19:31:26.914695 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:26.914671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:27.150778 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.150680 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6tch\" (UID: \"7ef36049-ec6f-4beb-9e2f-fbee5edb3920\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:27.150778 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.150726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tls-certs\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:27.153119 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.153090 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ef36049-ec6f-4beb-9e2f-fbee5edb3920-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6tch\" (UID: \"7ef36049-ec6f-4beb-9e2f-fbee5edb3920\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:27.153257 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.153228 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6fb9cb-80af-4da4-be5f-a7a4089505f8-tls-certs\") pod \"kube-auth-proxy-cff79d5b6-t5f5h\" (UID: \"be6fb9cb-80af-4da4-be5f-a7a4089505f8\") " pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:27.250674 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.250642 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" event={"ID":"54717a5e-11ed-46fb-98a4-a7613ba540d5","Type":"ContainerStarted","Data":"b1a1243ced70e35a979f32874aac818c07ea04b80427495a7b119fda71592572"} Jul 24 19:31:27.310586 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.310549 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" Jul 24 19:31:27.355062 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.355025 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:27.470760 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.455103 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h"] Jul 24 19:31:27.482341 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:31:27.482308 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6fb9cb_80af_4da4_be5f_a7a4089505f8.slice/crio-68c27e27177679d53623f2304dcb5d47d0ed0750c618fab8ae02f8836aee1483 WatchSource:0}: Error finding container 68c27e27177679d53623f2304dcb5d47d0ed0750c618fab8ae02f8836aee1483: Status 404 returned error can't find the container with id 68c27e27177679d53623f2304dcb5d47d0ed0750c618fab8ae02f8836aee1483 Jul 24 19:31:27.527307 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.527280 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch"] Jul 24 19:31:27.529728 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:31:27.529709 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7ef36049_ec6f_4beb_9e2f_fbee5edb3920.slice/crio-c8c4c4f8d1afa511d32564cc7c87f9c344fa651c679b018354eed49af4c9559a WatchSource:0}: Error finding container c8c4c4f8d1afa511d32564cc7c87f9c344fa651c679b018354eed49af4c9559a: Status 404 returned error can't find the container with id c8c4c4f8d1afa511d32564cc7c87f9c344fa651c679b018354eed49af4c9559a Jul 24 19:31:27.578341 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.578297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:27.693176 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.693087 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:27.811812 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:27.811772 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:28.257337 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:28.257304 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" event={"ID":"7ef36049-ec6f-4beb-9e2f-fbee5edb3920","Type":"ContainerStarted","Data":"c8c4c4f8d1afa511d32564cc7c87f9c344fa651c679b018354eed49af4c9559a"} Jul 24 19:31:28.258655 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:28.258629 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" event={"ID":"be6fb9cb-80af-4da4-be5f-a7a4089505f8","Type":"ContainerStarted","Data":"68c27e27177679d53623f2304dcb5d47d0ed0750c618fab8ae02f8836aee1483"} Jul 24 19:31:28.571231 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:28.571146 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:28.692537 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:28.692499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:29.832813 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:29.832781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:30.271401 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:30.271364 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" event={"ID":"54717a5e-11ed-46fb-98a4-a7613ba540d5","Type":"ContainerStarted","Data":"25b83979f43ea6987c1990479279dae1433e393d0a30e9c47cbb62007e495ed2"} Jul 24 19:31:30.271613 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:30.271484 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:31:30.288523 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:30.288353 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" podStartSLOduration=1.5106100150000001 podStartE2EDuration="4.288335991s" totalImagesPullingTime="2.777725976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:30.287707472 +0000 UTC m=+320.383196680" watchObservedRunningTime="2026-07-24 19:31:30.288335991 +0000 UTC m=+320.383825200" Jul 24 19:31:31.277399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:31.277286 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" event={"ID":"7ef36049-ec6f-4beb-9e2f-fbee5edb3920","Type":"ContainerStarted","Data":"15bb12f6b421f216e73960727660040992237b6e93e6ed11f92c45f3e1fdf168"} Jul 24 19:31:31.277938 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:31.277440 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:31:31.278908 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:31.278881 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" event={"ID":"be6fb9cb-80af-4da4-be5f-a7a4089505f8","Type":"ContainerStarted","Data":"36dedec524cf21f21c6f4d1cc7ac0fb221fe07bc6a5c0dc43a867cef5a605b68"} Jul 24 19:31:31.293594 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:31.293548 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" podStartSLOduration=1.815130965 podStartE2EDuration="5.29353387s" totalImagesPullingTime="3.478402905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:31.292390879 +0000 UTC m=+321.387880086" watchObservedRunningTime="2026-07-24 19:31:31.29353387 +0000 UTC m=+321.389023055" Jul 24 19:31:31.308918 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:31.308766 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cff79d5b6-t5f5h" podStartSLOduration=1.78632608 podStartE2EDuration="5.308747666s" totalImagesPullingTime="3.522421586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:31.308547045 +0000 UTC m=+321.404036247" watchObservedRunningTime="2026-07-24 19:31:31.308747666 +0000 UTC m=+321.404236874" Jul 24 19:31:32.229842 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:32.229814 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-j7z9v" Jul 24 19:31:32.231463 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:32.231439 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-r82sc" Jul 24 19:31:33.212273 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.212239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:33.313569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.313538 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:33.872330 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.872296 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s"] Jul 24 19:31:33.875778 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.875758 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.878348 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.878323 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 19:31:33.878498 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.878352 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tcrjd\"" Jul 24 19:31:33.888112 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.888090 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s"] Jul 24 19:31:33.899834 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.899810 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:33.920959 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.920928 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921053 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.920969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921053 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.920996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921053 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.921044 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b07f51-050d-4875-859b-1d627ac8a572-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921196 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.921077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwc6\" (UniqueName: \"kubernetes.io/projected/d2b07f51-050d-4875-859b-1d627ac8a572-kube-api-access-2vwc6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921196 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.921135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921277 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.921197 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b07f51-050d-4875-859b-1d627ac8a572-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921277 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.921237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b07f51-050d-4875-859b-1d627ac8a572-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:33.921277 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:33.921263 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.021979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.021945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022113 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.021995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022113 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b07f51-050d-4875-859b-1d627ac8a572-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022113 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022051 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwc6\" (UniqueName: \"kubernetes.io/projected/d2b07f51-050d-4875-859b-1d627ac8a572-kube-api-access-2vwc6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022255 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022255 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b07f51-050d-4875-859b-1d627ac8a572-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022255 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b07f51-050d-4875-859b-1d627ac8a572-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022255 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022220 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022375 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022439 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022391 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022637 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022587 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022770 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022748 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.022991 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.022972 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b07f51-050d-4875-859b-1d627ac8a572-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.024879 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.024853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b07f51-050d-4875-859b-1d627ac8a572-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.025209 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.025176 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b07f51-050d-4875-859b-1d627ac8a572-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.032762 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.032702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b07f51-050d-4875-859b-1d627ac8a572-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.033395 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.033128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwc6\" (UniqueName: \"kubernetes.io/projected/d2b07f51-050d-4875-859b-1d627ac8a572-kube-api-access-2vwc6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s\" (UID: \"d2b07f51-050d-4875-859b-1d627ac8a572\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.097185 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.097151 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:34.185747 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.185682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:34.212916 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.212882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:34.324402 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:34.324370 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s"] Jul 24 19:31:34.326425 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:31:34.326402 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b07f51_050d_4875_859b_1d627ac8a572.slice/crio-906b4a9aed1354d985350653831c8fb1db837ce8173dcf9f9368f645bbc87ea4 WatchSource:0}: Error finding container 906b4a9aed1354d985350653831c8fb1db837ce8173dcf9f9368f645bbc87ea4: Status 404 returned error can't find the container with id 906b4a9aed1354d985350653831c8fb1db837ce8173dcf9f9368f645bbc87ea4 Jul 24 19:31:35.300265 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:35.300226 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" event={"ID":"d2b07f51-050d-4875-859b-1d627ac8a572","Type":"ContainerStarted","Data":"906b4a9aed1354d985350653831c8fb1db837ce8173dcf9f9368f645bbc87ea4"} Jul 24 19:31:37.300798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:37.300742 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:31:37.301095 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:37.300832 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:31:37.301095 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:37.300862 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:31:37.512328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:37.512288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:37.588724 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:37.588641 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:31:38.314987 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:38.314951 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" event={"ID":"d2b07f51-050d-4875-859b-1d627ac8a572","Type":"ContainerStarted","Data":"1300eb6ef585e1a9a3637019f48ff1679764073380101127e288cea241acf559"} Jul 24 19:31:38.335410 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:38.335368 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" podStartSLOduration=2.363826825 podStartE2EDuration="5.335353754s" totalImagesPullingTime="2.971526929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:38.334137893 +0000 UTC m=+328.429627100" watchObservedRunningTime="2026-07-24 19:31:38.335353754 +0000 UTC m=+328.430843015" Jul 24 19:31:39.186002 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:39.185963 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:39.190309 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:39.190283 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:39.320527 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:39.320496 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:31:39.321560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:31:39.321542 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s" Jul 24 19:32:01.283919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:01.283889 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vgqt2" Jul 24 19:32:02.286971 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:02.286942 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6tch" Jul 24 19:32:16.736259 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.736218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:16.827306 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.827269 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:16.855988 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.855957 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5pkp"] Jul 24 19:32:16.866485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.866445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" Jul 24 19:32:16.869032 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.868946 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5pkp"] Jul 24 19:32:16.869178 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.869149 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-rrst6\"" Jul 24 19:32:16.869668 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.869486 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 19:32:16.870217 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.870157 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 19:32:16.905316 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.905285 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:16.986075 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:16.986038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrkh\" (UniqueName: \"kubernetes.io/projected/73921fc0-c430-4b04-820a-6e92a0adadd0-kube-api-access-xvrkh\") pod \"kuadrant-operator-catalog-d5pkp\" (UID: \"73921fc0-c430-4b04-820a-6e92a0adadd0\") " pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" Jul 24 19:32:17.087172 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.087077 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrkh\" (UniqueName: \"kubernetes.io/projected/73921fc0-c430-4b04-820a-6e92a0adadd0-kube-api-access-xvrkh\") pod \"kuadrant-operator-catalog-d5pkp\" (UID: \"73921fc0-c430-4b04-820a-6e92a0adadd0\") " pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" Jul 24 19:32:17.096014 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.095989 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrkh\" (UniqueName: \"kubernetes.io/projected/73921fc0-c430-4b04-820a-6e92a0adadd0-kube-api-access-xvrkh\") pod \"kuadrant-operator-catalog-d5pkp\" (UID: \"73921fc0-c430-4b04-820a-6e92a0adadd0\") " pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" Jul 24 19:32:17.177616 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.177582 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" Jul 24 19:32:17.222591 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.222557 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5pkp"] Jul 24 19:32:17.315801 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.315774 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5pkp"] Jul 24 19:32:17.319045 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:32:17.319017 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73921fc0_c430_4b04_820a_6e92a0adadd0.slice/crio-4c6d0f0d6a43a4de2132933efeabd617e1cb59cf4a4e2374f9d4cb01d71f2848 WatchSource:0}: Error finding container 4c6d0f0d6a43a4de2132933efeabd617e1cb59cf4a4e2374f9d4cb01d71f2848: Status 404 returned error can't find the container with id 4c6d0f0d6a43a4de2132933efeabd617e1cb59cf4a4e2374f9d4cb01d71f2848 Jul 24 19:32:17.434154 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.434077 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g5j2r"] Jul 24 19:32:17.439206 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.439188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:17.445846 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.445822 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g5j2r"] Jul 24 19:32:17.498687 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.498658 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" event={"ID":"73921fc0-c430-4b04-820a-6e92a0adadd0","Type":"ContainerStarted","Data":"4c6d0f0d6a43a4de2132933efeabd617e1cb59cf4a4e2374f9d4cb01d71f2848"} Jul 24 19:32:17.592119 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.592097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwklw\" (UniqueName: \"kubernetes.io/projected/c10c45e4-b62e-4a06-a0f2-42e3f58d9720-kube-api-access-bwklw\") pod \"kuadrant-operator-catalog-g5j2r\" (UID: \"c10c45e4-b62e-4a06-a0f2-42e3f58d9720\") " pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:17.693131 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.693048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwklw\" (UniqueName: \"kubernetes.io/projected/c10c45e4-b62e-4a06-a0f2-42e3f58d9720-kube-api-access-bwklw\") pod \"kuadrant-operator-catalog-g5j2r\" (UID: \"c10c45e4-b62e-4a06-a0f2-42e3f58d9720\") " pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:17.702633 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.702601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:17.704137 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.704118 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwklw\" (UniqueName: \"kubernetes.io/projected/c10c45e4-b62e-4a06-a0f2-42e3f58d9720-kube-api-access-bwklw\") pod \"kuadrant-operator-catalog-g5j2r\" (UID: \"c10c45e4-b62e-4a06-a0f2-42e3f58d9720\") " pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:17.749270 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.749245 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:17.867078 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.867044 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:17.887727 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:17.887704 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g5j2r"] Jul 24 19:32:17.890135 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:32:17.890111 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10c45e4_b62e_4a06_a0f2_42e3f58d9720.slice/crio-504bab665597734c37164512507ad8c1a71f51bf9a8a153ca586eb560d5f8638 WatchSource:0}: Error finding container 504bab665597734c37164512507ad8c1a71f51bf9a8a153ca586eb560d5f8638: Status 404 returned error can't find the container with id 504bab665597734c37164512507ad8c1a71f51bf9a8a153ca586eb560d5f8638 Jul 24 19:32:18.509574 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:18.509529 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" event={"ID":"c10c45e4-b62e-4a06-a0f2-42e3f58d9720","Type":"ContainerStarted","Data":"504bab665597734c37164512507ad8c1a71f51bf9a8a153ca586eb560d5f8638"} Jul 24 19:32:20.519957 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.519918 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" event={"ID":"c10c45e4-b62e-4a06-a0f2-42e3f58d9720","Type":"ContainerStarted","Data":"c0b2e9c55ac6e74351c6c403042d8e30f6b61abf4b2de7553ebb80dd4015e90d"} Jul 24 19:32:20.521267 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.521242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" event={"ID":"73921fc0-c430-4b04-820a-6e92a0adadd0","Type":"ContainerStarted","Data":"2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487"} Jul 24 19:32:20.521395 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.521347 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" podUID="73921fc0-c430-4b04-820a-6e92a0adadd0" containerName="registry-server" containerID="cri-o://2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487" gracePeriod=2 Jul 24 19:32:20.536524 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.536375 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" podStartSLOduration=1.972550208 podStartE2EDuration="3.536357892s" totalImagesPullingTime="1.563807684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:32:20.535049098 +0000 UTC m=+370.630538306" watchObservedRunningTime="2026-07-24 19:32:20.536357892 +0000 UTC m=+370.631847099" Jul 24 19:32:20.549848 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.549810 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" podStartSLOduration=2.415458853 podStartE2EDuration="4.54979712s" totalImagesPullingTime="2.134338267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:32:20.548973831 +0000 UTC m=+370.644463038" watchObservedRunningTime="2026-07-24 19:32:20.54979712 +0000 UTC m=+370.645286352" Jul 24 19:32:20.769274 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.769247 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" Jul 24 19:32:20.919529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.919436 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvrkh\" (UniqueName: \"kubernetes.io/projected/73921fc0-c430-4b04-820a-6e92a0adadd0-kube-api-access-xvrkh\") pod \"73921fc0-c430-4b04-820a-6e92a0adadd0\" (UID: \"73921fc0-c430-4b04-820a-6e92a0adadd0\") " Jul 24 19:32:20.921497 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:20.921456 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73921fc0-c430-4b04-820a-6e92a0adadd0-kube-api-access-xvrkh" (OuterVolumeSpecName: "kube-api-access-xvrkh") pod "73921fc0-c430-4b04-820a-6e92a0adadd0" (UID: "73921fc0-c430-4b04-820a-6e92a0adadd0"). InnerVolumeSpecName "kube-api-access-xvrkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:32:21.020716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.020683 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvrkh\" (UniqueName: \"kubernetes.io/projected/73921fc0-c430-4b04-820a-6e92a0adadd0-kube-api-access-xvrkh\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:32:21.527153 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.527116 2570 generic.go:356] "Generic (PLEG): container finished" podID="73921fc0-c430-4b04-820a-6e92a0adadd0" containerID="2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487" exitCode=0 Jul 24 19:32:21.527153 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.527156 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" Jul 24 19:32:21.527716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.527205 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" event={"ID":"73921fc0-c430-4b04-820a-6e92a0adadd0","Type":"ContainerDied","Data":"2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487"} Jul 24 19:32:21.527716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.527256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5pkp" event={"ID":"73921fc0-c430-4b04-820a-6e92a0adadd0","Type":"ContainerDied","Data":"4c6d0f0d6a43a4de2132933efeabd617e1cb59cf4a4e2374f9d4cb01d71f2848"} Jul 24 19:32:21.527716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.527277 2570 scope.go:117] "RemoveContainer" containerID="2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487" Jul 24 19:32:21.535835 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.535819 2570 scope.go:117] "RemoveContainer" containerID="2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487" Jul 24 19:32:21.536091 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:32:21.536071 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487\": container with ID starting with 2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487 not found: ID does not exist" containerID="2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487" Jul 24 19:32:21.536138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.536099 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487"} err="failed to get container status \"2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487\": rpc error: code = NotFound desc = could not find container \"2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487\": container with ID starting with 2a0fef5a918a0171ca3c5b29020d209f55d233c35cf083ac71142abbdc5dc487 not found: ID does not exist" Jul 24 19:32:21.550387 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.550363 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5pkp"] Jul 24 19:32:21.553983 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:21.553962 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5pkp"] Jul 24 19:32:22.580725 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:22.580694 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73921fc0-c430-4b04-820a-6e92a0adadd0" path="/var/lib/kubelet/pods/73921fc0-c430-4b04-820a-6e92a0adadd0/volumes" Jul 24 19:32:27.750085 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:27.750053 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:27.750513 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:27.750125 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:27.771637 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:27.771615 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:28.581097 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:28.581070 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-g5j2r" Jul 24 19:32:40.312718 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:40.312683 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:43.957340 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:43.957297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:45.071478 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.071435 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb"] Jul 24 19:32:45.071864 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.071820 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73921fc0-c430-4b04-820a-6e92a0adadd0" containerName="registry-server" Jul 24 19:32:45.071864 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.071834 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="73921fc0-c430-4b04-820a-6e92a0adadd0" containerName="registry-server" Jul 24 19:32:45.071935 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.071891 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="73921fc0-c430-4b04-820a-6e92a0adadd0" containerName="registry-server" Jul 24 19:32:45.075024 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.075001 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.077676 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.077654 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-twmhj\"" Jul 24 19:32:45.087937 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.087913 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb"] Jul 24 19:32:45.115661 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.115613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzr4c\" (UniqueName: \"kubernetes.io/projected/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-kube-api-access-kzr4c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.115840 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.115667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.216850 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.216810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzr4c\" (UniqueName: \"kubernetes.io/projected/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-kube-api-access-kzr4c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.216850 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.216851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.217204 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.217187 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.226213 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.226183 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzr4c\" (UniqueName: \"kubernetes.io/projected/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-kube-api-access-kzr4c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.385045 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.384947 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:45.522342 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.522318 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb"] Jul 24 19:32:45.524430 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.524403 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:45.525594 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:32:45.525567 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cea0338_c7f7_4e28_a41c_e9bbe55310ca.slice/crio-8c2cc240d4814af15e0443221a974e5d8ae1b71589fbb4568d1253c3e16ba332 WatchSource:0}: Error finding container 8c2cc240d4814af15e0443221a974e5d8ae1b71589fbb4568d1253c3e16ba332: Status 404 returned error can't find the container with id 8c2cc240d4814af15e0443221a974e5d8ae1b71589fbb4568d1253c3e16ba332 Jul 24 19:32:45.636541 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:45.636427 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" event={"ID":"9cea0338-c7f7-4e28-a41c-e9bbe55310ca","Type":"ContainerStarted","Data":"8c2cc240d4814af15e0443221a974e5d8ae1b71589fbb4568d1253c3e16ba332"} Jul 24 19:32:46.124294 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:46.124257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:47.390258 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:47.390219 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:48.941478 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:48.941443 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:49.040000 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:49.039956 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:50.503747 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.503714 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp"] Jul 24 19:32:50.507574 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.507553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" Jul 24 19:32:50.510651 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.510628 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zk5lr\"" Jul 24 19:32:50.510971 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.510941 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 19:32:50.525781 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.525759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp"] Jul 24 19:32:50.562214 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.562182 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rv7m\" (UniqueName: \"kubernetes.io/projected/f234cac5-12df-4873-b0b4-245d6bcdabe8-kube-api-access-5rv7m\") pod \"dns-operator-controller-manager-864dd78c4c-6kzsp\" (UID: \"f234cac5-12df-4873-b0b4-245d6bcdabe8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" Jul 24 19:32:50.663449 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.663407 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rv7m\" (UniqueName: \"kubernetes.io/projected/f234cac5-12df-4873-b0b4-245d6bcdabe8-kube-api-access-5rv7m\") pod \"dns-operator-controller-manager-864dd78c4c-6kzsp\" (UID: \"f234cac5-12df-4873-b0b4-245d6bcdabe8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" Jul 24 19:32:50.676200 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.676172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rv7m\" (UniqueName: \"kubernetes.io/projected/f234cac5-12df-4873-b0b4-245d6bcdabe8-kube-api-access-5rv7m\") pod \"dns-operator-controller-manager-864dd78c4c-6kzsp\" (UID: \"f234cac5-12df-4873-b0b4-245d6bcdabe8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" Jul 24 19:32:50.820079 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:50.819990 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" Jul 24 19:32:51.128360 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:51.128264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:51.607404 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:51.607380 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp"] Jul 24 19:32:51.609444 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:32:51.609417 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf234cac5_12df_4873_b0b4_245d6bcdabe8.slice/crio-124ea6f35afcbb71df76dd4fc669c464712c39813b0dff1f091ee6be66bfba81 WatchSource:0}: Error finding container 124ea6f35afcbb71df76dd4fc669c464712c39813b0dff1f091ee6be66bfba81: Status 404 returned error can't find the container with id 124ea6f35afcbb71df76dd4fc669c464712c39813b0dff1f091ee6be66bfba81 Jul 24 19:32:51.668144 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:51.668114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" event={"ID":"9cea0338-c7f7-4e28-a41c-e9bbe55310ca","Type":"ContainerStarted","Data":"7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc"} Jul 24 19:32:51.668293 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:51.668277 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:32:51.669405 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:51.669386 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" event={"ID":"f234cac5-12df-4873-b0b4-245d6bcdabe8","Type":"ContainerStarted","Data":"124ea6f35afcbb71df76dd4fc669c464712c39813b0dff1f091ee6be66bfba81"} Jul 24 19:32:51.686583 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:51.686541 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" podStartSLOduration=0.67888241 podStartE2EDuration="6.686529904s" totalImagesPullingTime="6.007647494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:32:51.685213593 +0000 UTC m=+401.780702801" watchObservedRunningTime="2026-07-24 19:32:51.686529904 +0000 UTC m=+401.782019111" Jul 24 19:32:52.603480 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.603433 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9"] Jul 24 19:32:52.607002 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.606980 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:32:52.609938 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.609916 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fx6vl\"" Jul 24 19:32:52.621302 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.621282 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9"] Jul 24 19:32:52.683913 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.683877 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgk2x\" (UniqueName: \"kubernetes.io/projected/cf1d75d1-7f52-4e69-b005-b50e0742aec3-kube-api-access-cgk2x\") pod \"limitador-operator-controller-manager-859d7b78bd-n59n9\" (UID: \"cf1d75d1-7f52-4e69-b005-b50e0742aec3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:32:52.785485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.785444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgk2x\" (UniqueName: \"kubernetes.io/projected/cf1d75d1-7f52-4e69-b005-b50e0742aec3-kube-api-access-cgk2x\") pod \"limitador-operator-controller-manager-859d7b78bd-n59n9\" (UID: \"cf1d75d1-7f52-4e69-b005-b50e0742aec3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:32:52.801268 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.801237 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgk2x\" (UniqueName: \"kubernetes.io/projected/cf1d75d1-7f52-4e69-b005-b50e0742aec3-kube-api-access-cgk2x\") pod \"limitador-operator-controller-manager-859d7b78bd-n59n9\" (UID: \"cf1d75d1-7f52-4e69-b005-b50e0742aec3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:32:52.921659 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.921580 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:32:52.958281 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:52.958236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:53.066996 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:53.066964 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9"] Jul 24 19:32:53.369546 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:32:53.369515 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1d75d1_7f52_4e69_b005_b50e0742aec3.slice/crio-671e758ab361dfdab2fdf0f6ec63d683274908e02c4d38d8ecf6abd8611ecb83 WatchSource:0}: Error finding container 671e758ab361dfdab2fdf0f6ec63d683274908e02c4d38d8ecf6abd8611ecb83: Status 404 returned error can't find the container with id 671e758ab361dfdab2fdf0f6ec63d683274908e02c4d38d8ecf6abd8611ecb83 Jul 24 19:32:53.688521 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:53.688417 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" event={"ID":"f234cac5-12df-4873-b0b4-245d6bcdabe8","Type":"ContainerStarted","Data":"f3f688a1a1f9788a34823313562b923bf322d959fb2435741073914178d12229"} Jul 24 19:32:53.689040 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:53.688522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" Jul 24 19:32:53.689787 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:53.689763 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" event={"ID":"cf1d75d1-7f52-4e69-b005-b50e0742aec3","Type":"ContainerStarted","Data":"671e758ab361dfdab2fdf0f6ec63d683274908e02c4d38d8ecf6abd8611ecb83"} Jul 24 19:32:53.712744 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:53.712696 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" podStartSLOduration=1.9004709370000001 podStartE2EDuration="3.71268092s" totalImagesPullingTime="1.812209983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:32:53.711296629 +0000 UTC m=+403.806785849" watchObservedRunningTime="2026-07-24 19:32:53.71268092 +0000 UTC m=+403.808170127" Jul 24 19:32:54.433553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.433511 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5"] Jul 24 19:32:54.437668 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.437639 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.438183 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.438096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:54.440132 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.440088 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 19:32:54.440293 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.440262 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 19:32:54.440562 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.440541 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h25wg\"" Jul 24 19:32:54.445131 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.445106 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5"] Jul 24 19:32:54.502956 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.502922 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nz2k\" (UniqueName: \"kubernetes.io/projected/f374077a-a01c-4188-a847-e9bafc7d5d5a-kube-api-access-4nz2k\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.503127 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.502994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f374077a-a01c-4188-a847-e9bafc7d5d5a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.503127 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.503045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f374077a-a01c-4188-a847-e9bafc7d5d5a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.555417 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.555379 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:54.604377 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.604340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f374077a-a01c-4188-a847-e9bafc7d5d5a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.604625 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.604526 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nz2k\" (UniqueName: \"kubernetes.io/projected/f374077a-a01c-4188-a847-e9bafc7d5d5a-kube-api-access-4nz2k\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.604625 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.604586 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f374077a-a01c-4188-a847-e9bafc7d5d5a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.605222 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.605199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f374077a-a01c-4188-a847-e9bafc7d5d5a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.607877 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.607850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f374077a-a01c-4188-a847-e9bafc7d5d5a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.613155 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.613120 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nz2k\" (UniqueName: \"kubernetes.io/projected/f374077a-a01c-4188-a847-e9bafc7d5d5a-kube-api-access-4nz2k\") pod \"kuadrant-console-plugin-66b5774985-96jc5\" (UID: \"f374077a-a01c-4188-a847-e9bafc7d5d5a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.724642 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.724531 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:54.751749 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.751714 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" Jul 24 19:32:54.868490 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.866031 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:32:54.933703 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:54.933678 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5"] Jul 24 19:32:55.116007 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:32:55.115962 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf374077a_a01c_4188_a847_e9bafc7d5d5a.slice/crio-e169452fbec6f5b0818c79fc751167af6efee96cb946e9be122ff10744159efe WatchSource:0}: Error finding container e169452fbec6f5b0818c79fc751167af6efee96cb946e9be122ff10744159efe: Status 404 returned error can't find the container with id e169452fbec6f5b0818c79fc751167af6efee96cb946e9be122ff10744159efe Jul 24 19:32:55.700204 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:55.700171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" event={"ID":"cf1d75d1-7f52-4e69-b005-b50e0742aec3","Type":"ContainerStarted","Data":"6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1"} Jul 24 19:32:55.700413 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:55.700246 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:32:55.701351 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:55.701331 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" event={"ID":"f374077a-a01c-4188-a847-e9bafc7d5d5a","Type":"ContainerStarted","Data":"e169452fbec6f5b0818c79fc751167af6efee96cb946e9be122ff10744159efe"} Jul 24 19:32:55.719186 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:32:55.719143 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" podStartSLOduration=1.92678633 podStartE2EDuration="3.719130272s" totalImagesPullingTime="1.792343942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:32:55.717287271 +0000 UTC m=+405.812776479" watchObservedRunningTime="2026-07-24 19:32:55.719130272 +0000 UTC m=+405.814619478" Jul 24 19:33:02.677382 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:02.677351 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:04.006280 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.006241 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:04.101513 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.101457 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:04.532496 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.532451 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb"] Jul 24 19:33:04.532695 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.532666 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" containerName="manager" containerID="cri-o://7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc" gracePeriod=2 Jul 24 19:33:04.540266 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.540235 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb"] Jul 24 19:33:04.544455 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:04.544415 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\": pods \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:04.551075 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.551046 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9"] Jul 24 19:33:04.552044 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.552006 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" containerName="manager" containerID="cri-o://6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1" gracePeriod=2 Jul 24 19:33:04.553671 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:04.553619 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:04.553790 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.553747 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:04.553961 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.553934 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg"] Jul 24 19:33:04.554393 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.554372 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" containerName="manager" Jul 24 19:33:04.554393 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.554393 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" containerName="manager" Jul 24 19:33:04.554553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.554510 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" containerName="manager" Jul 24 19:33:04.559992 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.559817 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.562808 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.562752 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9"] Jul 24 19:33:04.569079 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:04.568954 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"cf1d75d1-7f52-4e69-b005-b50e0742aec3\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-n59n9\": pods \"limitador-operator-controller-manager-859d7b78bd-n59n9\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:04.571595 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:04.571266 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:04.572776 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.572751 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg"] Jul 24 19:33:04.574192 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:04.573883 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n59n9\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:04.593685 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.593658 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7"] Jul 24 19:33:04.594233 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.594212 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" containerName="manager" Jul 24 19:33:04.594334 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.594237 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" containerName="manager" Jul 24 19:33:04.594392 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.594352 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" containerName="manager" Jul 24 19:33:04.597824 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.597802 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" Jul 24 19:33:04.607594 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.606956 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7"] Jul 24 19:33:04.696549 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.696522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6kzsp" Jul 24 19:33:04.702319 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.702298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjj5f\" (UniqueName: \"kubernetes.io/projected/03c79a14-1772-4ea4-8013-dea36ac2449a-kube-api-access-wjj5f\") pod \"limitador-operator-controller-manager-859d7b78bd-56mj7\" (UID: \"03c79a14-1772-4ea4-8013-dea36ac2449a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" Jul 24 19:33:04.702447 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.702332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7flhg\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.702544 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.702520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdm7\" (UniqueName: \"kubernetes.io/projected/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-kube-api-access-8jdm7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7flhg\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.803043 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.802962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdm7\" (UniqueName: \"kubernetes.io/projected/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-kube-api-access-8jdm7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7flhg\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.803194 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.803039 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjj5f\" (UniqueName: \"kubernetes.io/projected/03c79a14-1772-4ea4-8013-dea36ac2449a-kube-api-access-wjj5f\") pod \"limitador-operator-controller-manager-859d7b78bd-56mj7\" (UID: \"03c79a14-1772-4ea4-8013-dea36ac2449a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" Jul 24 19:33:04.803194 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.803071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7flhg\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.803582 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.803549 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7flhg\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.821875 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.821850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjj5f\" (UniqueName: \"kubernetes.io/projected/03c79a14-1772-4ea4-8013-dea36ac2449a-kube-api-access-wjj5f\") pod \"limitador-operator-controller-manager-859d7b78bd-56mj7\" (UID: \"03c79a14-1772-4ea4-8013-dea36ac2449a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" Jul 24 19:33:04.822209 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.822187 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdm7\" (UniqueName: \"kubernetes.io/projected/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-kube-api-access-8jdm7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7flhg\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.967974 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.967559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:04.975171 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:04.975143 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" Jul 24 19:33:06.395489 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:06.395431 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:06.495549 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:06.495511 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:11.204266 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:11.204232 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:11.289601 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:11.289567 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:11.412141 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:11.412105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:11.960660 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:11.960620 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fd4b7575f-ghkjw"] Jul 24 19:33:12.017384 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.017346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:12.025880 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.025847 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd4b7575f-ghkjw"] Jul 24 19:33:12.026050 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.025981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.028712 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.028688 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 19:33:12.028854 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.028837 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lbnbp\"" Jul 24 19:33:12.029186 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.029166 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 19:33:12.030746 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.029794 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 19:33:12.030746 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.029836 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 19:33:12.030746 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.030123 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 19:33:12.034576 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.034449 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 19:33:12.172308 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.172271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-config\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.172529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.172328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-service-ca\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.172529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.172359 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-trusted-ca-bundle\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.172529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.172389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx6tw\" (UniqueName: \"kubernetes.io/projected/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-kube-api-access-cx6tw\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.172529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.172485 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-serving-cert\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.172529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.172526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-oauth-serving-cert\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.172729 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.172598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-oauth-config\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.274072 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.274032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-serving-cert\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.274589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.274081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-oauth-serving-cert\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.274589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.274156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-oauth-config\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.274589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.274230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-config\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.274589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.274277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-service-ca\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.274589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.274309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-trusted-ca-bundle\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.274589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.274340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx6tw\" (UniqueName: \"kubernetes.io/projected/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-kube-api-access-cx6tw\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.275981 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.275828 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-trusted-ca-bundle\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.276123 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.276034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-service-ca\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.276432 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.276406 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-config\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.276768 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.276745 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-oauth-serving-cert\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.277721 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.277697 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-serving-cert\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.277872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.277853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-console-oauth-config\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.285690 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.285670 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx6tw\" (UniqueName: \"kubernetes.io/projected/00615fdb-ab53-46f0-9aba-4ecc4c9ccc00-kube-api-access-cx6tw\") pod \"console-5fd4b7575f-ghkjw\" (UID: \"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00\") " pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:12.337397 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:12.337361 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:14.998374 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:14.998335 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:15.099406 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:15.099349 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:17.200222 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.200198 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg"] Jul 24 19:33:17.233839 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:33:17.233798 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a27372_0b8b_4488_b2a1_3fcf7eaffbf6.slice/crio-a8d5dcabf2548c6c048d73b87e0b7d2f89fe7f4c3729f07567008807d545ea79 WatchSource:0}: Error finding container a8d5dcabf2548c6c048d73b87e0b7d2f89fe7f4c3729f07567008807d545ea79: Status 404 returned error can't find the container with id a8d5dcabf2548c6c048d73b87e0b7d2f89fe7f4c3729f07567008807d545ea79 Jul 24 19:33:17.238729 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.238710 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:17.240734 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.240709 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n59n9\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:17.254216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.254193 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:17.256384 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.256356 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n59n9\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:17.258350 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.258325 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:17.323374 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.323346 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgk2x\" (UniqueName: \"kubernetes.io/projected/cf1d75d1-7f52-4e69-b005-b50e0742aec3-kube-api-access-cgk2x\") pod \"cf1d75d1-7f52-4e69-b005-b50e0742aec3\" (UID: \"cf1d75d1-7f52-4e69-b005-b50e0742aec3\") " Jul 24 19:33:17.323553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.323420 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzr4c\" (UniqueName: \"kubernetes.io/projected/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-kube-api-access-kzr4c\") pod \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " Jul 24 19:33:17.323553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.323495 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-extensions-socket-volume\") pod \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\" (UID: \"9cea0338-c7f7-4e28-a41c-e9bbe55310ca\") " Jul 24 19:33:17.324009 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.323986 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9cea0338-c7f7-4e28-a41c-e9bbe55310ca" (UID: "9cea0338-c7f7-4e28-a41c-e9bbe55310ca"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:33:17.325441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.325413 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-kube-api-access-kzr4c" (OuterVolumeSpecName: "kube-api-access-kzr4c") pod "9cea0338-c7f7-4e28-a41c-e9bbe55310ca" (UID: "9cea0338-c7f7-4e28-a41c-e9bbe55310ca"). InnerVolumeSpecName "kube-api-access-kzr4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:33:17.325441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.325426 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1d75d1-7f52-4e69-b005-b50e0742aec3-kube-api-access-cgk2x" (OuterVolumeSpecName: "kube-api-access-cgk2x") pod "cf1d75d1-7f52-4e69-b005-b50e0742aec3" (UID: "cf1d75d1-7f52-4e69-b005-b50e0742aec3"). InnerVolumeSpecName "kube-api-access-cgk2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:33:17.424403 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.424363 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgk2x\" (UniqueName: \"kubernetes.io/projected/cf1d75d1-7f52-4e69-b005-b50e0742aec3-kube-api-access-cgk2x\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:33:17.424403 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.424403 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzr4c\" (UniqueName: \"kubernetes.io/projected/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-kube-api-access-kzr4c\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:33:17.424403 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.424418 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cea0338-c7f7-4e28-a41c-e9bbe55310ca-extensions-socket-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:33:17.427120 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.427088 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7"] Jul 24 19:33:17.430572 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.430546 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd4b7575f-ghkjw"] Jul 24 19:33:17.431043 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:33:17.431021 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c79a14_1772_4ea4_8013_dea36ac2449a.slice/crio-4a2c3bbd25d763c9328a7be158fb89978e8b534ef3b32d02c37cd5b5af3d830f WatchSource:0}: Error finding container 4a2c3bbd25d763c9328a7be158fb89978e8b534ef3b32d02c37cd5b5af3d830f: Status 404 returned error can't find the container with id 4a2c3bbd25d763c9328a7be158fb89978e8b534ef3b32d02c37cd5b5af3d830f Jul 24 19:33:17.432289 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:33:17.432265 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00615fdb_ab53_46f0_9aba_4ecc4c9ccc00.slice/crio-f987025d5e19bcc07edcd711e06dbc3896aee591445dbe690c126c8203cb1014 WatchSource:0}: Error finding container f987025d5e19bcc07edcd711e06dbc3896aee591445dbe690c126c8203cb1014: Status 404 returned error can't find the container with id f987025d5e19bcc07edcd711e06dbc3896aee591445dbe690c126c8203cb1014 Jul 24 19:33:17.830234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.830139 2570 generic.go:356] "Generic (PLEG): container finished" podID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" containerID="7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc" exitCode=0 Jul 24 19:33:17.830234 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.830190 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:17.830445 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.830243 2570 scope.go:117] "RemoveContainer" containerID="7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc" Jul 24 19:33:17.832139 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.832103 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" event={"ID":"f374077a-a01c-4188-a847-e9bafc7d5d5a","Type":"ContainerStarted","Data":"4d0cd7da1cc5e2cd96fbdcb9960c7346029012d0c80ff893e9654bf3b7d36731"} Jul 24 19:33:17.832803 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.832762 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n59n9\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:17.834615 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.834589 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:17.834838 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.834818 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4b7575f-ghkjw" event={"ID":"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00","Type":"ContainerStarted","Data":"3a67f206596e6f141cf6c657b9dc257eb8ae21314d696134b21b35e45a9c01f8"} Jul 24 19:33:17.834901 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.834880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4b7575f-ghkjw" event={"ID":"00615fdb-ab53-46f0-9aba-4ecc4c9ccc00","Type":"ContainerStarted","Data":"f987025d5e19bcc07edcd711e06dbc3896aee591445dbe690c126c8203cb1014"} Jul 24 19:33:17.836366 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.836337 2570 generic.go:356] "Generic (PLEG): container finished" podID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" containerID="6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1" exitCode=0 Jul 24 19:33:17.836509 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.836375 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:17.836509 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.836491 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n59n9\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:17.838243 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.838211 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:17.838578 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.838551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" event={"ID":"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6","Type":"ContainerStarted","Data":"b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32"} Jul 24 19:33:17.838674 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.838600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" event={"ID":"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6","Type":"ContainerStarted","Data":"a8d5dcabf2548c6c048d73b87e0b7d2f89fe7f4c3729f07567008807d545ea79"} Jul 24 19:33:17.838674 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.838630 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:17.840519 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.840498 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" event={"ID":"03c79a14-1772-4ea4-8013-dea36ac2449a","Type":"ContainerStarted","Data":"abad36f63a7aecb60f95a25214fde9869bedad3f24e1ce29b0d8ef0ea4bde8e2"} Jul 24 19:33:17.840708 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.840683 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" Jul 24 19:33:17.840822 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.840718 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" event={"ID":"03c79a14-1772-4ea4-8013-dea36ac2449a","Type":"ContainerStarted","Data":"4a2c3bbd25d763c9328a7be158fb89978e8b534ef3b32d02c37cd5b5af3d830f"} Jul 24 19:33:17.841882 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.841863 2570 scope.go:117] "RemoveContainer" containerID="7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc" Jul 24 19:33:17.842154 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.842133 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc\": container with ID starting with 7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc not found: ID does not exist" containerID="7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc" Jul 24 19:33:17.842231 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.842159 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc"} err="failed to get container status \"7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc\": rpc error: code = NotFound desc = could not find container \"7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc\": container with ID starting with 7c92eab214a36a0f75040d1b5645822c5879b479e110f2c60a1968defc9319cc not found: ID does not exist" Jul 24 19:33:17.842231 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.842175 2570 scope.go:117] "RemoveContainer" containerID="6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1" Jul 24 19:33:17.851162 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.851145 2570 scope.go:117] "RemoveContainer" containerID="6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1" Jul 24 19:33:17.851417 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.851401 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1\": container with ID starting with 6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1 not found: ID does not exist" containerID="6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1" Jul 24 19:33:17.851463 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.851425 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1"} err="failed to get container status \"6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1\": rpc error: code = NotFound desc = could not find container \"6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1\": container with ID starting with 6570bd6e4c6e97e463ddd7d3b41b831b10928a355493c2b991f557d978fa39f1 not found: ID does not exist" Jul 24 19:33:17.855684 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.855645 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-96jc5" podStartSLOduration=1.843805635 podStartE2EDuration="23.855632077s" totalImagesPullingTime="22.011826442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:33:17.85472647 +0000 UTC m=+427.950215672" watchObservedRunningTime="2026-07-24 19:33:17.855632077 +0000 UTC m=+427.951121283" Jul 24 19:33:17.870988 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.870937 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" podStartSLOduration=13.870920487 podStartE2EDuration="13.870920487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:33:17.870070255 +0000 UTC m=+427.965559462" watchObservedRunningTime="2026-07-24 19:33:17.870920487 +0000 UTC m=+427.966409694" Jul 24 19:33:17.887734 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.887680 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fd4b7575f-ghkjw" podStartSLOduration=6.8876589809999995 podStartE2EDuration="6.887658981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:33:17.887180417 +0000 UTC m=+427.982669627" watchObservedRunningTime="2026-07-24 19:33:17.887658981 +0000 UTC m=+427.983148189" Jul 24 19:33:17.903921 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:17.903867 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" podStartSLOduration=13.903846509 podStartE2EDuration="13.903846509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:33:17.902824878 +0000 UTC m=+427.998314085" watchObservedRunningTime="2026-07-24 19:33:17.903846509 +0000 UTC m=+427.999335717" Jul 24 19:33:17.904627 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.904596 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n59n9\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n59n9" Jul 24 19:33:17.906327 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:17.906298 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xqpzb\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xqpzb" Jul 24 19:33:18.581251 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:18.581220 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cea0338-c7f7-4e28-a41c-e9bbe55310ca" path="/var/lib/kubelet/pods/9cea0338-c7f7-4e28-a41c-e9bbe55310ca/volumes" Jul 24 19:33:18.581612 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:18.581568 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf1d75d1-7f52-4e69-b005-b50e0742aec3" path="/var/lib/kubelet/pods/cf1d75d1-7f52-4e69-b005-b50e0742aec3/volumes" Jul 24 19:33:22.337975 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:22.337942 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:22.338338 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:22.338000 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:22.342779 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:22.342748 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:22.872215 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:22.872182 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fd4b7575f-ghkjw" Jul 24 19:33:22.873838 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:22.873816 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg"] Jul 24 19:33:22.874017 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:22.873981 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" podUID="70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" containerName="manager" containerID="cri-o://b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32" gracePeriod=10 Jul 24 19:33:22.876178 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:22.876156 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:23.046111 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.046078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:23.130032 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.129971 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:23.144573 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.144549 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:23.170081 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.170055 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-extensions-socket-volume\") pod \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " Jul 24 19:33:23.170268 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.170161 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jdm7\" (UniqueName: \"kubernetes.io/projected/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-kube-api-access-8jdm7\") pod \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\" (UID: \"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6\") " Jul 24 19:33:23.170528 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.170498 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" (UID: "70a27372-0b8b-4488-b2a1-3fcf7eaffbf6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:33:23.172432 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.172406 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-kube-api-access-8jdm7" (OuterVolumeSpecName: "kube-api-access-8jdm7") pod "70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" (UID: "70a27372-0b8b-4488-b2a1-3fcf7eaffbf6"). InnerVolumeSpecName "kube-api-access-8jdm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:33:23.244483 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.244428 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:23.271489 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.271437 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-extensions-socket-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:33:23.271489 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.271463 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jdm7\" (UniqueName: \"kubernetes.io/projected/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6-kube-api-access-8jdm7\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:33:23.345933 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.345884 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:23.610197 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.610157 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:23.719401 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.719362 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:23.874622 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.874525 2570 generic.go:356] "Generic (PLEG): container finished" podID="70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" containerID="b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32" exitCode=0 Jul 24 19:33:23.874622 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.874584 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" Jul 24 19:33:23.874622 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.874606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" event={"ID":"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6","Type":"ContainerDied","Data":"b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32"} Jul 24 19:33:23.874895 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.874660 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg" event={"ID":"70a27372-0b8b-4488-b2a1-3fcf7eaffbf6","Type":"ContainerDied","Data":"a8d5dcabf2548c6c048d73b87e0b7d2f89fe7f4c3729f07567008807d545ea79"} Jul 24 19:33:23.874895 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.874684 2570 scope.go:117] "RemoveContainer" containerID="b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32" Jul 24 19:33:23.884504 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.884462 2570 scope.go:117] "RemoveContainer" containerID="b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32" Jul 24 19:33:23.884778 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:23.884757 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32\": container with ID starting with b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32 not found: ID does not exist" containerID="b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32" Jul 24 19:33:23.884856 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.884789 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32"} err="failed to get container status \"b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32\": rpc error: code = NotFound desc = could not find container \"b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32\": container with ID starting with b7973dcd20b712682df36ded44c38567c79d722c6ae1434c9ba2351b0c61ea32 not found: ID does not exist" Jul 24 19:33:23.897460 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.897431 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg"] Jul 24 19:33:23.904145 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:23.904122 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7flhg"] Jul 24 19:33:24.580923 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:24.580886 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" path="/var/lib/kubelet/pods/70a27372-0b8b-4488-b2a1-3fcf7eaffbf6/volumes" Jul 24 19:33:26.057924 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:26.057884 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:26.125817 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:26.125781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:28.850256 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:28.850223 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-56mj7" Jul 24 19:33:30.781711 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:30.781674 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:30.882874 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:30.882838 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:34.989705 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:34.989608 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:35.046493 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:35.046431 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:39.205202 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.205169 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h"] Jul 24 19:33:39.205742 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.205558 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" containerName="manager" Jul 24 19:33:39.205742 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.205570 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" containerName="manager" Jul 24 19:33:39.205742 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.205640 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a27372-0b8b-4488-b2a1-3fcf7eaffbf6" containerName="manager" Jul 24 19:33:39.230007 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.229944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:39.254458 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.254429 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h"] Jul 24 19:33:39.254608 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.254544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.257203 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.257180 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-k2gmg\"" Jul 24 19:33:39.318405 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318377 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318537 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318410 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318537 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318441 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vr8j\" (UniqueName: \"kubernetes.io/projected/375d0858-f3b5-4c3c-9003-181bb349d96a-kube-api-access-2vr8j\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318537 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318659 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318659 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375d0858-f3b5-4c3c-9003-181bb349d96a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318731 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318770 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318747 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.318911 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.318790 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.419875 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.419834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.419875 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.419881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.419913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.419957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.419975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.419995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vr8j\" (UniqueName: \"kubernetes.io/projected/375d0858-f3b5-4c3c-9003-181bb349d96a-kube-api-access-2vr8j\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420066 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375d0858-f3b5-4c3c-9003-181bb349d96a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420521 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420303 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420521 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420331 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420521 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420402 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420679 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420625 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.420885 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.420867 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375d0858-f3b5-4c3c-9003-181bb349d96a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.422616 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.422590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.422718 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.422627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.427755 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.427731 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375d0858-f3b5-4c3c-9003-181bb349d96a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.427869 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.427852 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vr8j\" (UniqueName: \"kubernetes.io/projected/375d0858-f3b5-4c3c-9003-181bb349d96a-kube-api-access-2vr8j\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hcz6h\" (UID: \"375d0858-f3b5-4c3c-9003-181bb349d96a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.565260 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.565231 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:39.644421 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.644360 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:39.730513 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.730490 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h"] Jul 24 19:33:39.733441 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:33:39.733411 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375d0858_f3b5_4c3c_9003_181bb349d96a.slice/crio-9352a0c89ce78a0a51d40ba837bed770f42041fcfb383460075b45e4660eacd1 WatchSource:0}: Error finding container 9352a0c89ce78a0a51d40ba837bed770f42041fcfb383460075b45e4660eacd1: Status 404 returned error can't find the container with id 9352a0c89ce78a0a51d40ba837bed770f42041fcfb383460075b45e4660eacd1 Jul 24 19:33:39.735584 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.735552 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:33:39.735659 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.735617 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:33:39.735659 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.735643 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:33:39.955959 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.955920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" event={"ID":"375d0858-f3b5-4c3c-9003-181bb349d96a","Type":"ContainerStarted","Data":"8655513fa9ce2767da2ca40cc878aa2c4d67c92b475877267ccd299b138d9343"} Jul 24 19:33:39.956155 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.956016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" event={"ID":"375d0858-f3b5-4c3c-9003-181bb349d96a","Type":"ContainerStarted","Data":"9352a0c89ce78a0a51d40ba837bed770f42041fcfb383460075b45e4660eacd1"} Jul 24 19:33:39.976308 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:39.976264 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" podStartSLOduration=0.976248064 podStartE2EDuration="976.248064ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:33:39.974602198 +0000 UTC m=+450.070091409" watchObservedRunningTime="2026-07-24 19:33:39.976248064 +0000 UTC m=+450.071737272" Jul 24 19:33:40.565429 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:40.565398 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:40.570045 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:40.570025 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:40.961521 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:40.961428 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:40.962373 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:40.962350 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:33:40.985630 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:40.985599 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-htvzc"] Jul 24 19:33:40.992321 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:40.992272 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:41.062275 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.062240 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-htvzc"] Jul 24 19:33:41.062445 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.062373 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.065086 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.064856 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 19:33:41.084740 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.084711 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-htvzc"] Jul 24 19:33:41.097122 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.097088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:41.135253 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.135228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9dcm\" (UniqueName: \"kubernetes.io/projected/a88edc66-3b3a-4731-b217-12c262698626-kube-api-access-n9dcm\") pod \"limitador-limitador-64c94fd64f-htvzc\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.135360 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.135276 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a88edc66-3b3a-4731-b217-12c262698626-config-file\") pod \"limitador-limitador-64c94fd64f-htvzc\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.196051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.196018 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:41.235853 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.235777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9dcm\" (UniqueName: \"kubernetes.io/projected/a88edc66-3b3a-4731-b217-12c262698626-kube-api-access-n9dcm\") pod \"limitador-limitador-64c94fd64f-htvzc\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.235853 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.235824 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a88edc66-3b3a-4731-b217-12c262698626-config-file\") pod \"limitador-limitador-64c94fd64f-htvzc\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.236407 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.236387 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a88edc66-3b3a-4731-b217-12c262698626-config-file\") pod \"limitador-limitador-64c94fd64f-htvzc\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.243948 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.243921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9dcm\" (UniqueName: \"kubernetes.io/projected/a88edc66-3b3a-4731-b217-12c262698626-kube-api-access-n9dcm\") pod \"limitador-limitador-64c94fd64f-htvzc\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.377013 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.376973 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:41.555040 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.555017 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-htvzc"] Jul 24 19:33:41.557504 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:33:41.557457 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88edc66_3b3a_4731_b217_12c262698626.slice/crio-af37229b6d6b77e95e4ce99408d542387e04269e65913c15a0b47bf353e257cd WatchSource:0}: Error finding container af37229b6d6b77e95e4ce99408d542387e04269e65913c15a0b47bf353e257cd: Status 404 returned error can't find the container with id af37229b6d6b77e95e4ce99408d542387e04269e65913c15a0b47bf353e257cd Jul 24 19:33:41.819796 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.819711 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:41.877297 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.877266 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:41.967693 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:41.967655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" event={"ID":"a88edc66-3b3a-4731-b217-12c262698626","Type":"ContainerStarted","Data":"af37229b6d6b77e95e4ce99408d542387e04269e65913c15a0b47bf353e257cd"} Jul 24 19:33:45.386680 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:45.386639 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:45.442106 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:45.442041 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:45.991834 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:45.991799 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:46.092396 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:46.092351 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:47.001093 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:47.001056 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" event={"ID":"a88edc66-3b3a-4731-b217-12c262698626","Type":"ContainerStarted","Data":"1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a"} Jul 24 19:33:47.001664 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:47.001139 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:47.019314 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:47.019271 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" podStartSLOduration=2.4638305 podStartE2EDuration="7.019258153s" totalImagesPullingTime="4.555427653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:33:47.018985965 +0000 UTC m=+457.114475192" watchObservedRunningTime="2026-07-24 19:33:47.019258153 +0000 UTC m=+457.114747360" Jul 24 19:33:49.052653 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:49.052613 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:49.785179 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:49.785141 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:49.839840 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:49.839802 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:53.798992 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:53.798957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:53.896283 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:53.896244 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:55.395449 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:55.395406 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-htvzc"] Jul 24 19:33:55.395957 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:55.395628 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" podUID="a88edc66-3b3a-4731-b217-12c262698626" containerName="limitador" containerID="cri-o://1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a" gracePeriod=30 Jul 24 19:33:55.396559 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:55.396253 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:55.949528 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:55.949504 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:56.045893 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.045863 2570 generic.go:356] "Generic (PLEG): container finished" podID="a88edc66-3b3a-4731-b217-12c262698626" containerID="1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a" exitCode=0 Jul 24 19:33:56.046051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.045904 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" Jul 24 19:33:56.046051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.045947 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" event={"ID":"a88edc66-3b3a-4731-b217-12c262698626","Type":"ContainerDied","Data":"1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a"} Jul 24 19:33:56.046051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.046001 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-htvzc" event={"ID":"a88edc66-3b3a-4731-b217-12c262698626","Type":"ContainerDied","Data":"af37229b6d6b77e95e4ce99408d542387e04269e65913c15a0b47bf353e257cd"} Jul 24 19:33:56.046051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.046030 2570 scope.go:117] "RemoveContainer" containerID="1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a" Jul 24 19:33:56.054716 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.054700 2570 scope.go:117] "RemoveContainer" containerID="1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a" Jul 24 19:33:56.054976 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:33:56.054958 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a\": container with ID starting with 1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a not found: ID does not exist" containerID="1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a" Jul 24 19:33:56.055027 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.054984 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a"} err="failed to get container status \"1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a\": rpc error: code = NotFound desc = could not find container \"1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a\": container with ID starting with 1b1876c9b8652b9f78ce3510f5ea9157c19a94369a19aca55315fec335126b7a not found: ID does not exist" Jul 24 19:33:56.069270 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.069251 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9dcm\" (UniqueName: \"kubernetes.io/projected/a88edc66-3b3a-4731-b217-12c262698626-kube-api-access-n9dcm\") pod \"a88edc66-3b3a-4731-b217-12c262698626\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " Jul 24 19:33:56.069345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.069296 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a88edc66-3b3a-4731-b217-12c262698626-config-file\") pod \"a88edc66-3b3a-4731-b217-12c262698626\" (UID: \"a88edc66-3b3a-4731-b217-12c262698626\") " Jul 24 19:33:56.069678 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.069657 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88edc66-3b3a-4731-b217-12c262698626-config-file" (OuterVolumeSpecName: "config-file") pod "a88edc66-3b3a-4731-b217-12c262698626" (UID: "a88edc66-3b3a-4731-b217-12c262698626"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:33:56.071430 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.071405 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88edc66-3b3a-4731-b217-12c262698626-kube-api-access-n9dcm" (OuterVolumeSpecName: "kube-api-access-n9dcm") pod "a88edc66-3b3a-4731-b217-12c262698626" (UID: "a88edc66-3b3a-4731-b217-12c262698626"). InnerVolumeSpecName "kube-api-access-n9dcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:33:56.170705 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.170679 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9dcm\" (UniqueName: \"kubernetes.io/projected/a88edc66-3b3a-4731-b217-12c262698626-kube-api-access-n9dcm\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:33:56.170705 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.170703 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a88edc66-3b3a-4731-b217-12c262698626-config-file\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:33:56.367519 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.367487 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-htvzc"] Jul 24 19:33:56.371519 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.371489 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-htvzc"] Jul 24 19:33:56.581111 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:56.581075 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a88edc66-3b3a-4731-b217-12c262698626" path="/var/lib/kubelet/pods/a88edc66-3b3a-4731-b217-12c262698626/volumes" Jul 24 19:33:57.097208 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.097155 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:57.194665 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.194629 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:57.558052 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.558024 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx"] Jul 24 19:33:57.558399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.558386 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a88edc66-3b3a-4731-b217-12c262698626" containerName="limitador" Jul 24 19:33:57.558454 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.558401 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a88edc66-3b3a-4731-b217-12c262698626" containerName="limitador" Jul 24 19:33:57.558514 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.558487 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a88edc66-3b3a-4731-b217-12c262698626" containerName="limitador" Jul 24 19:33:57.562564 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.562544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:57.565259 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.565234 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 19:33:57.565391 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.565312 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nrjx9\"" Jul 24 19:33:57.566272 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.566250 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 19:33:57.566399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.566308 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 19:33:57.569827 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.569803 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx"] Jul 24 19:33:57.605291 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.605253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:33:57.683337 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.683308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb9b9575-1402-4b68-928f-81862b667417-data\") pod \"postgres-69fb4594cf-hm2vx\" (UID: \"bb9b9575-1402-4b68-928f-81862b667417\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:57.683493 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.683389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jrdw\" (UniqueName: \"kubernetes.io/projected/bb9b9575-1402-4b68-928f-81862b667417-kube-api-access-7jrdw\") pod \"postgres-69fb4594cf-hm2vx\" (UID: \"bb9b9575-1402-4b68-928f-81862b667417\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:57.784175 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.784144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jrdw\" (UniqueName: \"kubernetes.io/projected/bb9b9575-1402-4b68-928f-81862b667417-kube-api-access-7jrdw\") pod \"postgres-69fb4594cf-hm2vx\" (UID: \"bb9b9575-1402-4b68-928f-81862b667417\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:57.784325 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.784202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb9b9575-1402-4b68-928f-81862b667417-data\") pod \"postgres-69fb4594cf-hm2vx\" (UID: \"bb9b9575-1402-4b68-928f-81862b667417\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:57.784579 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.784563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb9b9575-1402-4b68-928f-81862b667417-data\") pod \"postgres-69fb4594cf-hm2vx\" (UID: \"bb9b9575-1402-4b68-928f-81862b667417\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:57.791873 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.791845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jrdw\" (UniqueName: \"kubernetes.io/projected/bb9b9575-1402-4b68-928f-81862b667417-kube-api-access-7jrdw\") pod \"postgres-69fb4594cf-hm2vx\" (UID: \"bb9b9575-1402-4b68-928f-81862b667417\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:57.874563 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:57.874505 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:33:58.000858 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:58.000828 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx"] Jul 24 19:33:58.003004 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:33:58.002981 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9b9575_1402_4b68_928f_81862b667417.slice/crio-6fcf7e22ac989269235c6a5092612144c410eb05c05a3059150bf1b4affc81a8 WatchSource:0}: Error finding container 6fcf7e22ac989269235c6a5092612144c410eb05c05a3059150bf1b4affc81a8: Status 404 returned error can't find the container with id 6fcf7e22ac989269235c6a5092612144c410eb05c05a3059150bf1b4affc81a8 Jul 24 19:33:58.059339 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:33:58.059311 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" event={"ID":"bb9b9575-1402-4b68-928f-81862b667417","Type":"ContainerStarted","Data":"6fcf7e22ac989269235c6a5092612144c410eb05c05a3059150bf1b4affc81a8"} Jul 24 19:34:04.092843 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:04.092805 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" event={"ID":"bb9b9575-1402-4b68-928f-81862b667417","Type":"ContainerStarted","Data":"86a12447535578da76ca22fce84ac10628469a69d728d08f59e8ee3e70a2b9cb"} Jul 24 19:34:04.093209 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:04.092946 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:34:04.108806 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:04.108752 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" podStartSLOduration=1.6242965790000001 podStartE2EDuration="7.108734652s" totalImagesPullingTime="5.484438073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:34:04.107422475 +0000 UTC m=+474.202911683" watchObservedRunningTime="2026-07-24 19:34:04.108734652 +0000 UTC m=+474.204223859" Jul 24 19:34:10.127120 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:10.127089 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hm2vx" Jul 24 19:34:10.430125 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:10.430001 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:10.634857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:10.634826 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:16.725036 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.724990 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:16.740462 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.740422 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78bd64b69-jp8hr"] Jul 24 19:34:16.750022 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.749998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:16.757799 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.757775 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 19:34:16.758049 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.758022 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 19:34:16.758561 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.758538 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zsz96\"" Jul 24 19:34:16.760361 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.760338 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bd64b69-jp8hr"] Jul 24 19:34:16.822179 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.822136 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:16.872649 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.872624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert\") pod \"maas-controller-78bd64b69-jp8hr\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:16.872794 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.872675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f868t\" (UniqueName: \"kubernetes.io/projected/9ccb99c1-4a63-411b-9398-8ae14065e220-kube-api-access-f868t\") pod \"maas-controller-78bd64b69-jp8hr\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:16.882349 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.882324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:16.908534 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.908513 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f4c4bc9c-hvrhb"] Jul 24 19:34:16.912356 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.912339 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:16.922598 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.922542 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4c4bc9c-hvrhb"] Jul 24 19:34:16.974120 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.974095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert\") pod \"maas-controller-78bd64b69-jp8hr\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:16.974263 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.974132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert\") pod \"maas-controller-58f4c4bc9c-hvrhb\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:16.974263 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.974184 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f868t\" (UniqueName: \"kubernetes.io/projected/9ccb99c1-4a63-411b-9398-8ae14065e220-kube-api-access-f868t\") pod \"maas-controller-78bd64b69-jp8hr\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:16.974263 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:34:16.974247 2570 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 19:34:16.974263 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.974257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwc5l\" (UniqueName: \"kubernetes.io/projected/5b1113e5-7580-4f41-9991-21815d6f05ae-kube-api-access-xwc5l\") pod \"maas-controller-58f4c4bc9c-hvrhb\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:16.974463 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:34:16.974320 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert podName:9ccb99c1-4a63-411b-9398-8ae14065e220 nodeName:}" failed. No retries permitted until 2026-07-24 19:34:17.474296707 +0000 UTC m=+487.569785897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert") pod "maas-controller-78bd64b69-jp8hr" (UID: "9ccb99c1-4a63-411b-9398-8ae14065e220") : secret "maas-controller-webhook-cert" not found Jul 24 19:34:16.985181 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:16.985121 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f868t\" (UniqueName: \"kubernetes.io/projected/9ccb99c1-4a63-411b-9398-8ae14065e220-kube-api-access-f868t\") pod \"maas-controller-78bd64b69-jp8hr\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:17.074955 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.074927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert\") pod \"maas-controller-58f4c4bc9c-hvrhb\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:17.075049 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.075007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwc5l\" (UniqueName: \"kubernetes.io/projected/5b1113e5-7580-4f41-9991-21815d6f05ae-kube-api-access-xwc5l\") pod \"maas-controller-58f4c4bc9c-hvrhb\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:17.075093 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:34:17.075067 2570 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 19:34:17.075137 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:34:17.075127 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert podName:5b1113e5-7580-4f41-9991-21815d6f05ae nodeName:}" failed. No retries permitted until 2026-07-24 19:34:17.57511085 +0000 UTC m=+487.670600036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert") pod "maas-controller-58f4c4bc9c-hvrhb" (UID: "5b1113e5-7580-4f41-9991-21815d6f05ae") : secret "maas-controller-webhook-cert" not found Jul 24 19:34:17.084568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.084545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwc5l\" (UniqueName: \"kubernetes.io/projected/5b1113e5-7580-4f41-9991-21815d6f05ae-kube-api-access-xwc5l\") pod \"maas-controller-58f4c4bc9c-hvrhb\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:17.160311 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.160287 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:17.479220 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.479182 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert\") pod \"maas-controller-78bd64b69-jp8hr\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:17.481596 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.481577 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert\") pod \"maas-controller-78bd64b69-jp8hr\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:17.579817 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.579788 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert\") pod \"maas-controller-58f4c4bc9c-hvrhb\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:17.582041 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.582023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert\") pod \"maas-controller-58f4c4bc9c-hvrhb\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:17.662850 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.662828 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:17.796864 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.796837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bd64b69-jp8hr"] Jul 24 19:34:17.798396 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:34:17.798373 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccb99c1_4a63_411b_9398_8ae14065e220.slice/crio-57bb8206842fad228ee2868eeec947f8d8e43b58bbd0785a652eb54c8c5614dd WatchSource:0}: Error finding container 57bb8206842fad228ee2868eeec947f8d8e43b58bbd0785a652eb54c8c5614dd: Status 404 returned error can't find the container with id 57bb8206842fad228ee2868eeec947f8d8e43b58bbd0785a652eb54c8c5614dd Jul 24 19:34:17.823918 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.823895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:17.957387 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:17.957364 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4c4bc9c-hvrhb"] Jul 24 19:34:17.959360 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:34:17.959334 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1113e5_7580_4f41_9991_21815d6f05ae.slice/crio-795c8a6104856daf3066eccaa6342a01ce0e3dc188cf5143d3542247ddd36ef0 WatchSource:0}: Error finding container 795c8a6104856daf3066eccaa6342a01ce0e3dc188cf5143d3542247ddd36ef0: Status 404 returned error can't find the container with id 795c8a6104856daf3066eccaa6342a01ce0e3dc188cf5143d3542247ddd36ef0 Jul 24 19:34:18.164548 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:18.164451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" event={"ID":"5b1113e5-7580-4f41-9991-21815d6f05ae","Type":"ContainerStarted","Data":"795c8a6104856daf3066eccaa6342a01ce0e3dc188cf5143d3542247ddd36ef0"} Jul 24 19:34:18.165611 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:18.165588 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bd64b69-jp8hr" event={"ID":"9ccb99c1-4a63-411b-9398-8ae14065e220","Type":"ContainerStarted","Data":"57bb8206842fad228ee2868eeec947f8d8e43b58bbd0785a652eb54c8c5614dd"} Jul 24 19:34:20.619355 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:20.619318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:20.717201 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:20.717113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:21.187664 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:21.187616 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" event={"ID":"5b1113e5-7580-4f41-9991-21815d6f05ae","Type":"ContainerStarted","Data":"3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05"} Jul 24 19:34:21.187862 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:21.187692 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:21.189252 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:21.189224 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bd64b69-jp8hr" event={"ID":"9ccb99c1-4a63-411b-9398-8ae14065e220","Type":"ContainerStarted","Data":"4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad"} Jul 24 19:34:21.189372 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:21.189352 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:21.209977 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:21.209921 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" podStartSLOduration=2.71398386 podStartE2EDuration="5.20990842s" totalImagesPullingTime="2.49592456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:34:21.208396708 +0000 UTC m=+491.303885912" watchObservedRunningTime="2026-07-24 19:34:21.20990842 +0000 UTC m=+491.305397628" Jul 24 19:34:21.237156 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:21.237098 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78bd64b69-jp8hr" podStartSLOduration=2.586447817 podStartE2EDuration="5.237078456s" totalImagesPullingTime="2.650630639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:34:21.235740356 +0000 UTC m=+491.331229579" watchObservedRunningTime="2026-07-24 19:34:21.237078456 +0000 UTC m=+491.332567664" Jul 24 19:34:32.195977 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.195946 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:32.196530 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.196021 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:34:32.268933 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.268892 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78bd64b69-jp8hr"] Jul 24 19:34:32.269120 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.269078 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78bd64b69-jp8hr" podUID="9ccb99c1-4a63-411b-9398-8ae14065e220" containerName="manager" containerID="cri-o://4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad" gracePeriod=30 Jul 24 19:34:32.521990 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.521968 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:32.618872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.618846 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert\") pod \"9ccb99c1-4a63-411b-9398-8ae14065e220\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " Jul 24 19:34:32.619014 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.618874 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f868t\" (UniqueName: \"kubernetes.io/projected/9ccb99c1-4a63-411b-9398-8ae14065e220-kube-api-access-f868t\") pod \"9ccb99c1-4a63-411b-9398-8ae14065e220\" (UID: \"9ccb99c1-4a63-411b-9398-8ae14065e220\") " Jul 24 19:34:32.621155 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.621129 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccb99c1-4a63-411b-9398-8ae14065e220-kube-api-access-f868t" (OuterVolumeSpecName: "kube-api-access-f868t") pod "9ccb99c1-4a63-411b-9398-8ae14065e220" (UID: "9ccb99c1-4a63-411b-9398-8ae14065e220"). InnerVolumeSpecName "kube-api-access-f868t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:34:32.621155 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.621144 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9ccb99c1-4a63-411b-9398-8ae14065e220" (UID: "9ccb99c1-4a63-411b-9398-8ae14065e220"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 19:34:32.720091 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.720065 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ccb99c1-4a63-411b-9398-8ae14065e220-webhook-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:34:32.720219 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:32.720095 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f868t\" (UniqueName: \"kubernetes.io/projected/9ccb99c1-4a63-411b-9398-8ae14065e220-kube-api-access-f868t\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:34:33.251028 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.250992 2570 generic.go:356] "Generic (PLEG): container finished" podID="9ccb99c1-4a63-411b-9398-8ae14065e220" containerID="4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad" exitCode=0 Jul 24 19:34:33.251443 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.251038 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bd64b69-jp8hr" Jul 24 19:34:33.251443 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.251080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bd64b69-jp8hr" event={"ID":"9ccb99c1-4a63-411b-9398-8ae14065e220","Type":"ContainerDied","Data":"4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad"} Jul 24 19:34:33.251443 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.251132 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bd64b69-jp8hr" event={"ID":"9ccb99c1-4a63-411b-9398-8ae14065e220","Type":"ContainerDied","Data":"57bb8206842fad228ee2868eeec947f8d8e43b58bbd0785a652eb54c8c5614dd"} Jul 24 19:34:33.251443 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.251153 2570 scope.go:117] "RemoveContainer" containerID="4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad" Jul 24 19:34:33.260658 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.260643 2570 scope.go:117] "RemoveContainer" containerID="4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad" Jul 24 19:34:33.260883 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:34:33.260868 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad\": container with ID starting with 4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad not found: ID does not exist" containerID="4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad" Jul 24 19:34:33.260936 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.260892 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad"} err="failed to get container status \"4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad\": rpc error: code = NotFound desc = could not find container \"4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad\": container with ID starting with 4b445d001b4b129d56a01ee7ae636fa141e7b3fea87c99a43984f624556268ad not found: ID does not exist" Jul 24 19:34:33.271920 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.271893 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78bd64b69-jp8hr"] Jul 24 19:34:33.275868 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:33.275844 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78bd64b69-jp8hr"] Jul 24 19:34:34.581968 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:34.581932 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ccb99c1-4a63-411b-9398-8ae14065e220" path="/var/lib/kubelet/pods/9ccb99c1-4a63-411b-9398-8ae14065e220/volumes" Jul 24 19:34:53.676880 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:53.676830 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:54.153357 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.153318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:54.249296 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.249252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:54.821922 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.821886 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:54.879628 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.879597 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf"] Jul 24 19:34:54.879989 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.879977 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ccb99c1-4a63-411b-9398-8ae14065e220" containerName="manager" Jul 24 19:34:54.880034 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.879991 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ccb99c1-4a63-411b-9398-8ae14065e220" containerName="manager" Jul 24 19:34:54.880069 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.880055 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ccb99c1-4a63-411b-9398-8ae14065e220" containerName="manager" Jul 24 19:34:54.907925 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.907887 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf"] Jul 24 19:34:54.908099 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.907977 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:54.910721 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.910695 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 19:34:54.910865 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:54.910699 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z5t42\"" Jul 24 19:34:55.012421 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.012392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c30409-6ca8-4775-864f-b8f08ea55005-openshift-service-ca\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.012617 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.012442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5jd\" (UniqueName: \"kubernetes.io/projected/d3c30409-6ca8-4775-864f-b8f08ea55005-kube-api-access-sh5jd\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.012617 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.012463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c30409-6ca8-4775-864f-b8f08ea55005-maas-api-tls\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.070973 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.070931 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:55.113731 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.113652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c30409-6ca8-4775-864f-b8f08ea55005-openshift-service-ca\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.113731 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.113711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5jd\" (UniqueName: \"kubernetes.io/projected/d3c30409-6ca8-4775-864f-b8f08ea55005-kube-api-access-sh5jd\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.113731 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.113732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c30409-6ca8-4775-864f-b8f08ea55005-maas-api-tls\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.114348 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.114326 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c30409-6ca8-4775-864f-b8f08ea55005-openshift-service-ca\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.116021 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.115999 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c30409-6ca8-4775-864f-b8f08ea55005-maas-api-tls\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.121623 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.121594 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5jd\" (UniqueName: \"kubernetes.io/projected/d3c30409-6ca8-4775-864f-b8f08ea55005-kube-api-access-sh5jd\") pod \"maas-api-6896c585f8-sfgcf\" (UID: \"d3c30409-6ca8-4775-864f-b8f08ea55005\") " pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.221399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.221364 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:55.390119 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.390068 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf"] Jul 24 19:34:55.393507 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:34:55.393477 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c30409_6ca8_4775_864f_b8f08ea55005.slice/crio-ab337990e145e3242b3ec24e527c82c979857a8e9b6b90e6849534fc9c105380 WatchSource:0}: Error finding container ab337990e145e3242b3ec24e527c82c979857a8e9b6b90e6849534fc9c105380: Status 404 returned error can't find the container with id ab337990e145e3242b3ec24e527c82c979857a8e9b6b90e6849534fc9c105380 Jul 24 19:34:55.473849 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.473821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:55.563115 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.563071 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:34:55.565503 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.565458 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw"] Jul 24 19:34:55.570254 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.570238 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.572888 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.572870 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 19:34:55.579407 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.579379 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw"] Jul 24 19:34:55.618569 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.618543 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdzf\" (UniqueName: \"kubernetes.io/projected/fc9d3da8-dc69-4f32-bf97-762cf5140d3a-kube-api-access-4fdzf\") pod \"payload-pre-processing-687dbb6cf6-8fgfw\" (UID: \"fc9d3da8-dc69-4f32-bf97-762cf5140d3a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.618684 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.618664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9d3da8-dc69-4f32-bf97-762cf5140d3a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-8fgfw\" (UID: \"fc9d3da8-dc69-4f32-bf97-762cf5140d3a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.707449 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.707367 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-cxnzs"] Jul 24 19:34:55.711016 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.710998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:55.713299 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.713280 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-stj6v\"" Jul 24 19:34:55.720179 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.719287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9d3da8-dc69-4f32-bf97-762cf5140d3a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-8fgfw\" (UID: \"fc9d3da8-dc69-4f32-bf97-762cf5140d3a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.720179 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.719366 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdzf\" (UniqueName: \"kubernetes.io/projected/fc9d3da8-dc69-4f32-bf97-762cf5140d3a-kube-api-access-4fdzf\") pod \"payload-pre-processing-687dbb6cf6-8fgfw\" (UID: \"fc9d3da8-dc69-4f32-bf97-762cf5140d3a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.720370 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.720245 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9d3da8-dc69-4f32-bf97-762cf5140d3a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-8fgfw\" (UID: \"fc9d3da8-dc69-4f32-bf97-762cf5140d3a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.721930 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.721908 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-cxnzs"] Jul 24 19:34:55.729675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.729649 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdzf\" (UniqueName: \"kubernetes.io/projected/fc9d3da8-dc69-4f32-bf97-762cf5140d3a-kube-api-access-4fdzf\") pod \"payload-pre-processing-687dbb6cf6-8fgfw\" (UID: \"fc9d3da8-dc69-4f32-bf97-762cf5140d3a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.820183 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.820152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjks\" (UniqueName: \"kubernetes.io/projected/ecc30144-c524-45fc-8d8a-66e9ebef2c82-kube-api-access-dwjks\") pod \"payload-processing-5cc6cd444b-cxnzs\" (UID: \"ecc30144-c524-45fc-8d8a-66e9ebef2c82\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:55.820350 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.820222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc30144-c524-45fc-8d8a-66e9ebef2c82-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-cxnzs\" (UID: \"ecc30144-c524-45fc-8d8a-66e9ebef2c82\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:55.881553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.881526 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:34:55.921075 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.921046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc30144-c524-45fc-8d8a-66e9ebef2c82-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-cxnzs\" (UID: \"ecc30144-c524-45fc-8d8a-66e9ebef2c82\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:55.921215 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.921139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjks\" (UniqueName: \"kubernetes.io/projected/ecc30144-c524-45fc-8d8a-66e9ebef2c82-kube-api-access-dwjks\") pod \"payload-processing-5cc6cd444b-cxnzs\" (UID: \"ecc30144-c524-45fc-8d8a-66e9ebef2c82\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:55.921927 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.921896 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc30144-c524-45fc-8d8a-66e9ebef2c82-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-cxnzs\" (UID: \"ecc30144-c524-45fc-8d8a-66e9ebef2c82\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:55.929392 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:55.929372 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjks\" (UniqueName: \"kubernetes.io/projected/ecc30144-c524-45fc-8d8a-66e9ebef2c82-kube-api-access-dwjks\") pod \"payload-processing-5cc6cd444b-cxnzs\" (UID: \"ecc30144-c524-45fc-8d8a-66e9ebef2c82\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:56.022765 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:56.022734 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:34:56.204769 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:56.204276 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-cxnzs"] Jul 24 19:34:56.206442 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:34:56.206403 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc30144_c524_45fc_8d8a_66e9ebef2c82.slice/crio-47be5a708a4f0b093853b8df2dac0cc9d0ef0fda48ecb54b170413bd717d886c WatchSource:0}: Error finding container 47be5a708a4f0b093853b8df2dac0cc9d0ef0fda48ecb54b170413bd717d886c: Status 404 returned error can't find the container with id 47be5a708a4f0b093853b8df2dac0cc9d0ef0fda48ecb54b170413bd717d886c Jul 24 19:34:56.230173 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:56.230147 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw"] Jul 24 19:34:56.371433 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:56.371392 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" event={"ID":"ecc30144-c524-45fc-8d8a-66e9ebef2c82","Type":"ContainerStarted","Data":"47be5a708a4f0b093853b8df2dac0cc9d0ef0fda48ecb54b170413bd717d886c"} Jul 24 19:34:56.372762 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:56.372736 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" event={"ID":"fc9d3da8-dc69-4f32-bf97-762cf5140d3a","Type":"ContainerStarted","Data":"83f3d8cdaad0e5308e5e3569fd90475803b254fef8c3d3700deac0b0c2f2a6a9"} Jul 24 19:34:56.373993 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:56.373967 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" event={"ID":"d3c30409-6ca8-4775-864f-b8f08ea55005","Type":"ContainerStarted","Data":"ab337990e145e3242b3ec24e527c82c979857a8e9b6b90e6849534fc9c105380"} Jul 24 19:34:58.388138 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:58.388097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" event={"ID":"d3c30409-6ca8-4775-864f-b8f08ea55005","Type":"ContainerStarted","Data":"bea8ab154b0897afa84a4f59c03e1635caa6e437fbfbea3f4e2a6c72079d2b7e"} Jul 24 19:34:58.388634 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:58.388219 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:34:58.406772 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:34:58.406716 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" podStartSLOduration=2.091231386 podStartE2EDuration="4.40669537s" totalImagesPullingTime="2.315463984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:34:58.405608084 +0000 UTC m=+528.501097293" watchObservedRunningTime="2026-07-24 19:34:58.40669537 +0000 UTC m=+528.502184579" Jul 24 19:35:00.400177 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:00.400137 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" event={"ID":"ecc30144-c524-45fc-8d8a-66e9ebef2c82","Type":"ContainerStarted","Data":"d72756202b6d5b450a01d0974175de27df23394adac6c8830f216b74b0519127"} Jul 24 19:35:00.400645 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:00.400306 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:35:00.401625 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:00.401597 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" event={"ID":"fc9d3da8-dc69-4f32-bf97-762cf5140d3a","Type":"ContainerStarted","Data":"91ffcb45920dd7cb1476156fb123fc35df98d337a3e4e05c45ae51ab943748c0"} Jul 24 19:35:00.401752 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:00.401651 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:35:00.439782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:00.439729 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" podStartSLOduration=2.181064634 podStartE2EDuration="5.439711087s" totalImagesPullingTime="3.258646453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:35:00.43778304 +0000 UTC m=+530.533272248" watchObservedRunningTime="2026-07-24 19:35:00.439711087 +0000 UTC m=+530.535200293" Jul 24 19:35:00.473564 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:00.473503 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" podStartSLOduration=2.258767499 podStartE2EDuration="5.473459835s" totalImagesPullingTime="3.214692336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:35:00.472840026 +0000 UTC m=+530.568329233" watchObservedRunningTime="2026-07-24 19:35:00.473459835 +0000 UTC m=+530.568949053" Jul 24 19:35:03.419373 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:03.419287 2570 generic.go:356] "Generic (PLEG): container finished" podID="375d0858-f3b5-4c3c-9003-181bb349d96a" containerID="8655513fa9ce2767da2ca40cc878aa2c4d67c92b475877267ccd299b138d9343" exitCode=0 Jul 24 19:35:03.419793 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:03.419374 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" event={"ID":"375d0858-f3b5-4c3c-9003-181bb349d96a","Type":"ContainerDied","Data":"8655513fa9ce2767da2ca40cc878aa2c4d67c92b475877267ccd299b138d9343"} Jul 24 19:35:03.419793 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:03.419733 2570 scope.go:117] "RemoveContainer" containerID="8655513fa9ce2767da2ca40cc878aa2c4d67c92b475877267ccd299b138d9343" Jul 24 19:35:03.420801 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:03.420773 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:35:03.420864 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:03.420841 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:35:03.420900 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:03.420871 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:35:04.399391 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:04.399357 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6896c585f8-sfgcf" Jul 24 19:35:04.427237 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:04.427190 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" event={"ID":"375d0858-f3b5-4c3c-9003-181bb349d96a","Type":"ContainerStarted","Data":"3715f783e9e67d855f4e39e63563f8e392569a47b98025b949bc4b6c4c8d3681"} Jul 24 19:35:04.565648 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:04.565609 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:35:04.566273 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:04.566249 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hcz6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 19:35:04.566375 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:04.566321 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" podUID="375d0858-f3b5-4c3c-9003-181bb349d96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 19:35:05.392298 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:05.392257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:05.492844 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:05.492807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:05.566024 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:05.565991 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hcz6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 19:35:05.566199 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:05.566041 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" podUID="375d0858-f3b5-4c3c-9003-181bb349d96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 19:35:06.566129 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:06.566096 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hcz6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 19:35:06.566543 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:06.566173 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" podUID="375d0858-f3b5-4c3c-9003-181bb349d96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 19:35:07.565914 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:07.565879 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hcz6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 19:35:07.566091 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:07.565931 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" podUID="375d0858-f3b5-4c3c-9003-181bb349d96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 19:35:08.570061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:08.570031 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:35:08.570449 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:08.570393 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:35:08.571094 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:08.571076 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hcz6h" Jul 24 19:35:11.409589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:11.409557 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8fgfw" Jul 24 19:35:11.409975 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:11.409649 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-cxnzs" Jul 24 19:35:21.014085 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:21.014047 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:24.131869 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:24.131825 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:34.497907 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:34.497860 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:35.991228 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:35.991196 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq"] Jul 24 19:35:35.994944 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:35.994920 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:35.997556 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:35.997528 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 19:35:35.997677 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:35.997574 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 19:35:35.998556 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:35.998538 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mdztr\"" Jul 24 19:35:35.998646 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:35.998610 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 19:35:36.003902 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.003882 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq"] Jul 24 19:35:36.017843 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.017816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:36.101504 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.101452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.101690 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.101591 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.101690 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.101675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.101809 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.101716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2787d13f-2b4d-4659-a627-67b8ecfba54e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.101809 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.101741 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.101809 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.101763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkr4\" (UniqueName: \"kubernetes.io/projected/2787d13f-2b4d-4659-a627-67b8ecfba54e-kube-api-access-4mkr4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203202 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203202 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2787d13f-2b4d-4659-a627-67b8ecfba54e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203459 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203219 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203459 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkr4\" (UniqueName: \"kubernetes.io/projected/2787d13f-2b4d-4659-a627-67b8ecfba54e-kube-api-access-4mkr4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203459 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203459 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203353 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203713 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203661 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203713 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203681 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.203782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.203744 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.205678 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.205660 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2787d13f-2b4d-4659-a627-67b8ecfba54e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.205947 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.205931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2787d13f-2b4d-4659-a627-67b8ecfba54e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.210614 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.210596 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkr4\" (UniqueName: \"kubernetes.io/projected/2787d13f-2b4d-4659-a627-67b8ecfba54e-kube-api-access-4mkr4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrtbq\" (UID: \"2787d13f-2b4d-4659-a627-67b8ecfba54e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.306874 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.306807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:36.441060 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.441031 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq"] Jul 24 19:35:36.442717 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:35:36.442684 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2787d13f_2b4d_4659_a627_67b8ecfba54e.slice/crio-535c94cc0da4d56fb9928be45969d35c0a18c269cdcd069ebcf71bf6944de1c3 WatchSource:0}: Error finding container 535c94cc0da4d56fb9928be45969d35c0a18c269cdcd069ebcf71bf6944de1c3: Status 404 returned error can't find the container with id 535c94cc0da4d56fb9928be45969d35c0a18c269cdcd069ebcf71bf6944de1c3 Jul 24 19:35:36.597282 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:36.597198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" event={"ID":"2787d13f-2b4d-4659-a627-67b8ecfba54e","Type":"ContainerStarted","Data":"535c94cc0da4d56fb9928be45969d35c0a18c269cdcd069ebcf71bf6944de1c3"} Jul 24 19:35:37.604931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:37.604893 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" event={"ID":"2787d13f-2b4d-4659-a627-67b8ecfba54e","Type":"ContainerStarted","Data":"7c81c7fc4adc709a3ba5e2d6eb0964257af36ad7c929bc111a95c73a857e2da6"} Jul 24 19:35:37.622295 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:37.622239 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" podStartSLOduration=1.687326264 podStartE2EDuration="2.62222132s" totalImagesPullingTime="934.895056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:35:37.621847244 +0000 UTC m=+567.717336452" watchObservedRunningTime="2026-07-24 19:35:37.62222132 +0000 UTC m=+567.717710527" Jul 24 19:35:37.847763 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:37.847733 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:46.307325 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:46.307291 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:46.307792 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:46.307378 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:46.319859 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:46.319835 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:46.660307 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:46.660236 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrtbq" Jul 24 19:35:48.442932 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:48.442894 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:35:57.559907 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:35:57.559874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:36:02.073058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.073023 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp"] Jul 24 19:36:02.076926 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.076902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.081399 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.081380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 19:36:02.111966 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.111940 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp"] Jul 24 19:36:02.160579 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.160538 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:36:02.227600 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.227566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.227782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.227626 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46gfh\" (UniqueName: \"kubernetes.io/projected/caaa698c-1694-4088-a815-049aa4f3a707-kube-api-access-46gfh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.227782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.227687 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.227782 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.227752 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.227918 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.227796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.227918 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.227849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caaa698c-1694-4088-a815-049aa4f3a707-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328420 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328420 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328383 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caaa698c-1694-4088-a815-049aa4f3a707-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328420 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328733 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328453 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46gfh\" (UniqueName: \"kubernetes.io/projected/caaa698c-1694-4088-a815-049aa4f3a707-kube-api-access-46gfh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328733 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328733 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328851 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328803 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328922 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.328946 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.328939 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.330801 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.330784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caaa698c-1694-4088-a815-049aa4f3a707-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.331085 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.331068 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caaa698c-1694-4088-a815-049aa4f3a707-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.342125 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.342101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46gfh\" (UniqueName: \"kubernetes.io/projected/caaa698c-1694-4088-a815-049aa4f3a707-kube-api-access-46gfh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fkddp\" (UID: \"caaa698c-1694-4088-a815-049aa4f3a707\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.386931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.386909 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:02.529407 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.529378 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp"] Jul 24 19:36:02.530636 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:36:02.530594 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaa698c_1694_4088_a815_049aa4f3a707.slice/crio-94946b1bdcce2d6220c6d3033f64cdc6c977a60490cbe014efcd50bfae6c0620 WatchSource:0}: Error finding container 94946b1bdcce2d6220c6d3033f64cdc6c977a60490cbe014efcd50bfae6c0620: Status 404 returned error can't find the container with id 94946b1bdcce2d6220c6d3033f64cdc6c977a60490cbe014efcd50bfae6c0620 Jul 24 19:36:02.730495 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:02.730445 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" event={"ID":"caaa698c-1694-4088-a815-049aa4f3a707","Type":"ContainerStarted","Data":"94946b1bdcce2d6220c6d3033f64cdc6c977a60490cbe014efcd50bfae6c0620"} Jul 24 19:36:03.737851 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:03.737815 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" event={"ID":"caaa698c-1694-4088-a815-049aa4f3a707","Type":"ContainerStarted","Data":"b9be50815a16fb765666655c0e3c1fd70df4f0eb43a686832c365d61371735aa"} Jul 24 19:36:03.818551 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:03.818500 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" podStartSLOduration=1.643917805 podStartE2EDuration="1.818485977s" totalImagesPullingTime="174.568172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:36:03.815241722 +0000 UTC m=+593.910730927" watchObservedRunningTime="2026-07-24 19:36:03.818485977 +0000 UTC m=+593.913975177" Jul 24 19:36:06.417148 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:06.417115 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:36:10.360954 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:10.360913 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:36:10.364108 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:10.364089 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:36:12.387792 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:12.387759 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:12.388225 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:12.387820 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:12.400107 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:12.400083 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:12.795306 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:12.795274 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fkddp" Jul 24 19:36:17.539982 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:17.539944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:36:34.947344 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:36:34.947258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:37:33.832679 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:37:33.832642 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:37:33.930190 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:37:33.930147 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:37:34.993589 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:37:34.993537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:37:35.088699 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:37:35.088664 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:38:21.428817 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.428784 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f4c4bc9c-hvrhb"] Jul 24 19:38:21.429376 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.429052 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" podUID="5b1113e5-7580-4f41-9991-21815d6f05ae" containerName="manager" containerID="cri-o://3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05" gracePeriod=30 Jul 24 19:38:21.689354 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.689291 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:38:21.740917 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.740883 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwc5l\" (UniqueName: \"kubernetes.io/projected/5b1113e5-7580-4f41-9991-21815d6f05ae-kube-api-access-xwc5l\") pod \"5b1113e5-7580-4f41-9991-21815d6f05ae\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " Jul 24 19:38:21.741090 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.741036 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert\") pod \"5b1113e5-7580-4f41-9991-21815d6f05ae\" (UID: \"5b1113e5-7580-4f41-9991-21815d6f05ae\") " Jul 24 19:38:21.742950 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.742920 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1113e5-7580-4f41-9991-21815d6f05ae-kube-api-access-xwc5l" (OuterVolumeSpecName: "kube-api-access-xwc5l") pod "5b1113e5-7580-4f41-9991-21815d6f05ae" (UID: "5b1113e5-7580-4f41-9991-21815d6f05ae"). InnerVolumeSpecName "kube-api-access-xwc5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:38:21.743065 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.742959 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5b1113e5-7580-4f41-9991-21815d6f05ae" (UID: "5b1113e5-7580-4f41-9991-21815d6f05ae"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 19:38:21.842603 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.842572 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwc5l\" (UniqueName: \"kubernetes.io/projected/5b1113e5-7580-4f41-9991-21815d6f05ae-kube-api-access-xwc5l\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:38:21.842603 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:21.842601 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b1113e5-7580-4f41-9991-21815d6f05ae-webhook-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:38:22.441863 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.441829 2570 generic.go:356] "Generic (PLEG): container finished" podID="5b1113e5-7580-4f41-9991-21815d6f05ae" containerID="3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05" exitCode=0 Jul 24 19:38:22.442294 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.441875 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" Jul 24 19:38:22.442294 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.441913 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" event={"ID":"5b1113e5-7580-4f41-9991-21815d6f05ae","Type":"ContainerDied","Data":"3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05"} Jul 24 19:38:22.442294 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.441969 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4c4bc9c-hvrhb" event={"ID":"5b1113e5-7580-4f41-9991-21815d6f05ae","Type":"ContainerDied","Data":"795c8a6104856daf3066eccaa6342a01ce0e3dc188cf5143d3542247ddd36ef0"} Jul 24 19:38:22.442294 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.441996 2570 scope.go:117] "RemoveContainer" containerID="3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05" Jul 24 19:38:22.452183 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.452167 2570 scope.go:117] "RemoveContainer" containerID="3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05" Jul 24 19:38:22.452430 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:38:22.452410 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05\": container with ID starting with 3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05 not found: ID does not exist" containerID="3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05" Jul 24 19:38:22.452543 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.452438 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05"} err="failed to get container status \"3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05\": rpc error: code = NotFound desc = could not find container \"3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05\": container with ID starting with 3bb5900d7a804a6206f6deab4bc114db0455285f95312743231e2f2186aecf05 not found: ID does not exist" Jul 24 19:38:22.464661 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.464642 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f4c4bc9c-hvrhb"] Jul 24 19:38:22.469916 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.469891 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58f4c4bc9c-hvrhb"] Jul 24 19:38:22.581555 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:38:22.581528 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1113e5-7580-4f41-9991-21815d6f05ae" path="/var/lib/kubelet/pods/5b1113e5-7580-4f41-9991-21815d6f05ae/volumes" Jul 24 19:39:42.116320 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:39:42.116222 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:39:43.294051 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:39:43.294016 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:16.775723 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:16.775684 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:16.880485 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:16.880440 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:16.995971 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:16.995938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:22.798022 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:22.797986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:22.914436 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:22.914400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:23.001887 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:23.001857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:23.213951 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:23.213871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:29.247615 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:29.247583 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:40:29.360553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:40:29.360525 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:41:10.396356 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:41:10.396322 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:41:10.396910 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:41:10.396855 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:42:24.645518 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:42:24.645481 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:42:26.065737 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:42:26.065699 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:42:52.359696 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:42:52.359616 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:42:52.451248 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:42:52.451218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:43:48.348076 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:43:48.348034 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:46:10.427815 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:46:10.427782 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:46:10.428363 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:46:10.428091 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:46:17.652286 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:46:17.652253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:48:02.165276 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:48:02.165241 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:48:13.626561 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:48:13.626526 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:48:45.246841 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:48:45.246806 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:48:46.711048 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:48:46.711008 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:49:55.586458 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:49:55.586427 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:49:55.687056 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:49:55.687024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:50:37.051732 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:50:37.051699 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:50:53.133828 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:50:53.133788 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:50:53.233240 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:50:53.233207 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:51:10.465997 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:51:10.465964 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:51:10.467390 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:51:10.467366 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:52:00.631568 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:00.631461 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:24.134923 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:24.134881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:27.199494 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:27.199439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:27.296819 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:27.296776 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:27.666041 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:27.666009 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:33.299519 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.299458 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:33.359615 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.359585 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:33.523793 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.523766 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:33.816512 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.816478 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4"] Jul 24 19:52:33.816920 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.816907 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1113e5-7580-4f41-9991-21815d6f05ae" containerName="manager" Jul 24 19:52:33.816920 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.816921 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1113e5-7580-4f41-9991-21815d6f05ae" containerName="manager" Jul 24 19:52:33.817070 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.816992 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1113e5-7580-4f41-9991-21815d6f05ae" containerName="manager" Jul 24 19:52:33.820289 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.820269 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.822659 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.822636 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9bce955c-openshift-default-dockercfg-rj4k4\"" Jul 24 19:52:33.833346 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.833316 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:33.841790 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.841759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4"] Jul 24 19:52:33.919080 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-token\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919274 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-podinfo\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919274 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919129 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-data\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919274 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919153 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af18a66-0fdb-4655-82d9-5716c7e55dae-istiod-ca-cert\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919448 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-socket\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919448 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919303 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-envoy\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919448 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919335 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-certs\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919448 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-credential-socket\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:33.919448 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:33.919434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29mv\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-kube-api-access-r29mv\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.020541 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020506 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-token\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.020712 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-podinfo\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.020811 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-data\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.020849 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af18a66-0fdb-4655-82d9-5716c7e55dae-istiod-ca-cert\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.020888 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-socket\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.020923 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-envoy\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.020970 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-certs\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.021036 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.020980 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-credential-socket\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.021100 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.021035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29mv\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-kube-api-access-r29mv\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.021267 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.021243 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-data\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.021359 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.021287 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-socket\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.021409 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.021370 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-certs\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.021623 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.021597 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-credential-socket\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.021798 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.021688 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af18a66-0fdb-4655-82d9-5716c7e55dae-istiod-ca-cert\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.023119 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.023088 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-envoy\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.023206 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.023197 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-podinfo\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.028586 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.028560 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29mv\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-kube-api-access-r29mv\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.029396 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.029375 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-token\") pod \"e2e-ait-9bce955c-openshift-default-c5c585db-746n4\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.054677 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.054649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:34.131659 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.131571 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:34.270494 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.270447 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4"] Jul 24 19:52:34.272593 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:52:34.272555 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af18a66_0fdb_4655_82d9_5716c7e55dae.slice/crio-2f46446bd05cd779350d16d9192a7c7427d041e3a163fe3c6972008fcc310400 WatchSource:0}: Error finding container 2f46446bd05cd779350d16d9192a7c7427d041e3a163fe3c6972008fcc310400: Status 404 returned error can't find the container with id 2f46446bd05cd779350d16d9192a7c7427d041e3a163fe3c6972008fcc310400 Jul 24 19:52:34.274773 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.274738 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:52:34.274872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.274804 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:52:34.274872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.274831 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:52:34.830624 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.830584 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" event={"ID":"8af18a66-0fdb-4655-82d9-5716c7e55dae","Type":"ContainerStarted","Data":"26ad7e6e2f74a7f6e9d30603871b3d630c9f9bb86a504d4fd34546fb829b6460"} Jul 24 19:52:34.831029 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.830645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" event={"ID":"8af18a66-0fdb-4655-82d9-5716c7e55dae","Type":"ContainerStarted","Data":"2f46446bd05cd779350d16d9192a7c7427d041e3a163fe3c6972008fcc310400"} Jul 24 19:52:34.853793 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:34.853740 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" podStartSLOduration=1.8537239410000002 podStartE2EDuration="1.853723941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:52:34.85135052 +0000 UTC m=+1584.946839727" watchObservedRunningTime="2026-07-24 19:52:34.853723941 +0000 UTC m=+1584.949213150" Jul 24 19:52:35.132601 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:35.132511 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:35.137330 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:35.137304 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:35.835742 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:35.835698 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:35.836842 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:35.836822 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:36.288320 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:36.288282 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:36.369337 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:36.369304 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:36.445984 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:36.445935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:36.809284 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:36.809242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:37.058328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.058264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:37.164032 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.163950 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:52:37.172655 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.172625 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc"] Jul 24 19:52:37.176337 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.176321 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.179171 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.179153 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9bce955c\"" Jul 24 19:52:37.179322 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.179305 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9bce955c-dockercfg-q446r\"" Jul 24 19:52:37.192831 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.192806 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc"] Jul 24 19:52:37.254919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.254892 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xkzc\" (UniqueName: \"kubernetes.io/projected/adbfd853-1436-42b8-a6b6-e7505989f529-kube-api-access-2xkzc\") pod \"payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.255061 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.254941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbfd853-1436-42b8-a6b6-e7505989f529-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.356108 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.356073 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbfd853-1436-42b8-a6b6-e7505989f529-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.356304 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.356228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xkzc\" (UniqueName: \"kubernetes.io/projected/adbfd853-1436-42b8-a6b6-e7505989f529-kube-api-access-2xkzc\") pod \"payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.356763 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.356744 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbfd853-1436-42b8-a6b6-e7505989f529-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.363872 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.363856 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xkzc\" (UniqueName: \"kubernetes.io/projected/adbfd853-1436-42b8-a6b6-e7505989f529-kube-api-access-2xkzc\") pod \"payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.487560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.487533 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.629701 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.629664 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc"] Jul 24 19:52:37.631558 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:52:37.631521 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbfd853_1436_42b8_a6b6_e7505989f529.slice/crio-e24b6802f75aad8e6432757277d8ce6bf999d473c515e73b8ba2a256bfc77056 WatchSource:0}: Error finding container e24b6802f75aad8e6432757277d8ce6bf999d473c515e73b8ba2a256bfc77056: Status 404 returned error can't find the container with id e24b6802f75aad8e6432757277d8ce6bf999d473c515e73b8ba2a256bfc77056 Jul 24 19:52:37.849216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.849137 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" event={"ID":"adbfd853-1436-42b8-a6b6-e7505989f529","Type":"ContainerStarted","Data":"decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c"} Jul 24 19:52:37.849216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.849195 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" event={"ID":"adbfd853-1436-42b8-a6b6-e7505989f529","Type":"ContainerStarted","Data":"e24b6802f75aad8e6432757277d8ce6bf999d473c515e73b8ba2a256bfc77056"} Jul 24 19:52:37.849395 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.849219 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:37.866765 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:37.866715 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" podStartSLOduration=0.866701402 podStartE2EDuration="866.701402ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:52:37.863855704 +0000 UTC m=+1587.959344911" watchObservedRunningTime="2026-07-24 19:52:37.866701402 +0000 UTC m=+1587.962190611" Jul 24 19:52:48.856099 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:48.856055 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:52:51.757831 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.756630 2570 patch_prober.go:28] interesting pod/e2e-ait-9bce955c-openshift-default-c5c585db-746n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:55278->10.132.0.53:15021: read: connection reset by peer" start-of-body= Jul 24 19:52:51.757831 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.756687 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:55278->10.132.0.53:15021: read: connection reset by peer" Jul 24 19:52:51.929725 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.929655 2570 generic.go:356] "Generic (PLEG): container finished" podID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerID="26ad7e6e2f74a7f6e9d30603871b3d630c9f9bb86a504d4fd34546fb829b6460" exitCode=0 Jul 24 19:52:51.929725 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.929699 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" event={"ID":"8af18a66-0fdb-4655-82d9-5716c7e55dae","Type":"ContainerDied","Data":"26ad7e6e2f74a7f6e9d30603871b3d630c9f9bb86a504d4fd34546fb829b6460"} Jul 24 19:52:51.930077 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.930063 2570 scope.go:117] "RemoveContainer" containerID="26ad7e6e2f74a7f6e9d30603871b3d630c9f9bb86a504d4fd34546fb829b6460" Jul 24 19:52:51.931192 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.931157 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:52:51.931259 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.931236 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:52:51.931295 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:51.931276 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 19:52:52.940343 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:52.940296 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" event={"ID":"8af18a66-0fdb-4655-82d9-5716c7e55dae","Type":"ContainerStarted","Data":"f72e0d8094b8577c7b29851b520e5243724085e3e584b4cd3ab5d79d3e207a8c"} Jul 24 19:52:53.131894 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:53.131858 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:53.132512 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:53.132077 2570 patch_prober.go:28] interesting pod/e2e-ait-9bce955c-openshift-default-c5c585db-746n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 24 19:52:53.132512 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:53.132229 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 24 19:52:54.132855 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:54.132811 2570 patch_prober.go:28] interesting pod/e2e-ait-9bce955c-openshift-default-c5c585db-746n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 24 19:52:54.133365 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:54.132878 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 24 19:52:55.132636 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:55.132599 2570 patch_prober.go:28] interesting pod/e2e-ait-9bce955c-openshift-default-c5c585db-746n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 24 19:52:55.132907 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:55.132877 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 24 19:52:56.132433 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:56.132394 2570 patch_prober.go:28] interesting pod/e2e-ait-9bce955c-openshift-default-c5c585db-746n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 24 19:52:56.132768 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:56.132738 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 24 19:52:57.135310 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:57.135277 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:57.135740 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:57.135515 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:52:57.136207 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:52:57.136190 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:53:12.192379 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.192346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:12.224801 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.224764 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc"] Jul 24 19:53:12.225036 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.225004 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" podUID="adbfd853-1436-42b8-a6b6-e7505989f529" containerName="payload-pre-processing" containerID="cri-o://decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c" gracePeriod=30 Jul 24 19:53:12.501335 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.501310 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:53:12.599414 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.599383 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbfd853-1436-42b8-a6b6-e7505989f529-plugins-config-volume\") pod \"adbfd853-1436-42b8-a6b6-e7505989f529\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " Jul 24 19:53:12.599618 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.599418 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xkzc\" (UniqueName: \"kubernetes.io/projected/adbfd853-1436-42b8-a6b6-e7505989f529-kube-api-access-2xkzc\") pod \"adbfd853-1436-42b8-a6b6-e7505989f529\" (UID: \"adbfd853-1436-42b8-a6b6-e7505989f529\") " Jul 24 19:53:12.599883 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.599856 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbfd853-1436-42b8-a6b6-e7505989f529-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "adbfd853-1436-42b8-a6b6-e7505989f529" (UID: "adbfd853-1436-42b8-a6b6-e7505989f529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:53:12.601752 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.601723 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbfd853-1436-42b8-a6b6-e7505989f529-kube-api-access-2xkzc" (OuterVolumeSpecName: "kube-api-access-2xkzc") pod "adbfd853-1436-42b8-a6b6-e7505989f529" (UID: "adbfd853-1436-42b8-a6b6-e7505989f529"). InnerVolumeSpecName "kube-api-access-2xkzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:53:12.700919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.700882 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbfd853-1436-42b8-a6b6-e7505989f529-plugins-config-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:12.700919 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:12.700914 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xkzc\" (UniqueName: \"kubernetes.io/projected/adbfd853-1436-42b8-a6b6-e7505989f529-kube-api-access-2xkzc\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:13.054085 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.054051 2570 generic.go:356] "Generic (PLEG): container finished" podID="adbfd853-1436-42b8-a6b6-e7505989f529" containerID="decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c" exitCode=0 Jul 24 19:53:13.054285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.054096 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" Jul 24 19:53:13.054285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.054135 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" event={"ID":"adbfd853-1436-42b8-a6b6-e7505989f529","Type":"ContainerDied","Data":"decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c"} Jul 24 19:53:13.054285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.054185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc" event={"ID":"adbfd853-1436-42b8-a6b6-e7505989f529","Type":"ContainerDied","Data":"e24b6802f75aad8e6432757277d8ce6bf999d473c515e73b8ba2a256bfc77056"} Jul 24 19:53:13.054285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.054209 2570 scope.go:117] "RemoveContainer" containerID="decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c" Jul 24 19:53:13.063673 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.063657 2570 scope.go:117] "RemoveContainer" containerID="decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c" Jul 24 19:53:13.063905 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:53:13.063887 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c\": container with ID starting with decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c not found: ID does not exist" containerID="decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c" Jul 24 19:53:13.063954 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.063912 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c"} err="failed to get container status \"decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c\": rpc error: code = NotFound desc = could not find container \"decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c\": container with ID starting with decb90cdec3e4f22df1421b04dcade8dedc740771cb767f9de62204adb86e47c not found: ID does not exist" Jul 24 19:53:13.077357 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.077331 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc"] Jul 24 19:53:13.082968 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.082943 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bce955c-87f8d7ffb-m7dkc"] Jul 24 19:53:13.607916 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:13.607878 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:14.581675 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:14.581643 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbfd853-1436-42b8-a6b6-e7505989f529" path="/var/lib/kubelet/pods/adbfd853-1436-42b8-a6b6-e7505989f529/volumes" Jul 24 19:53:16.840223 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:16.840191 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:16.989237 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:16.989192 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4"] Jul 24 19:53:16.989440 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:16.989404 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" containerID="cri-o://f72e0d8094b8577c7b29851b520e5243724085e3e584b4cd3ab5d79d3e207a8c" gracePeriod=30 Jul 24 19:53:17.018024 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:17.017996 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:17.237847 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:17.237816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:17.471065 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:17.470976 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:17.570710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:17.570679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:17.696708 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:17.696679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:18.042967 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:18.042939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:18.211285 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:18.211256 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:22.103688 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.103659 2570 generic.go:356] "Generic (PLEG): container finished" podID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerID="f72e0d8094b8577c7b29851b520e5243724085e3e584b4cd3ab5d79d3e207a8c" exitCode=0 Jul 24 19:53:22.104082 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.103734 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" event={"ID":"8af18a66-0fdb-4655-82d9-5716c7e55dae","Type":"ContainerDied","Data":"f72e0d8094b8577c7b29851b520e5243724085e3e584b4cd3ab5d79d3e207a8c"} Jul 24 19:53:22.104082 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.103821 2570 scope.go:117] "RemoveContainer" containerID="26ad7e6e2f74a7f6e9d30603871b3d630c9f9bb86a504d4fd34546fb829b6460" Jul 24 19:53:22.242646 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.242622 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:53:22.295008 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.294925 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-credential-socket\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295008 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.294974 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-podinfo\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295008 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295008 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r29mv\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-kube-api-access-r29mv\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295034 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-data\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295077 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-envoy\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295127 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-certs\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295160 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af18a66-0fdb-4655-82d9-5716c7e55dae-istiod-ca-cert\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295216 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-socket\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295227 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:53:22.295278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295246 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-token\") pod \"8af18a66-0fdb-4655-82d9-5716c7e55dae\" (UID: \"8af18a66-0fdb-4655-82d9-5716c7e55dae\") " Jul 24 19:53:22.295639 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295613 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:53:22.295709 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295656 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:53:22.295709 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295670 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af18a66-0fdb-4655-82d9-5716c7e55dae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:53:22.295709 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.295699 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-credential-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.296035 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.296009 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-data" (OuterVolumeSpecName: "istio-data") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:53:22.297343 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.297316 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-kube-api-access-r29mv" (OuterVolumeSpecName: "kube-api-access-r29mv") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "kube-api-access-r29mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:53:22.297667 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.297635 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-token" (OuterVolumeSpecName: "istio-token") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:53:22.297747 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.297695 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 19:53:22.297747 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.297732 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8af18a66-0fdb-4655-82d9-5716c7e55dae" (UID: "8af18a66-0fdb-4655-82d9-5716c7e55dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:53:22.396605 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396570 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-data\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.396605 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396600 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-envoy\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.396605 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396610 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-certs\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.396857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396618 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af18a66-0fdb-4655-82d9-5716c7e55dae-istiod-ca-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.396857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396626 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af18a66-0fdb-4655-82d9-5716c7e55dae-workload-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.396857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396636 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-token\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.396857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396644 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af18a66-0fdb-4655-82d9-5716c7e55dae-istio-podinfo\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.396857 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.396655 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r29mv\" (UniqueName: \"kubernetes.io/projected/8af18a66-0fdb-4655-82d9-5716c7e55dae-kube-api-access-r29mv\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:53:22.869452 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.869361 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:22.977059 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:22.977018 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:23.110288 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.110245 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" event={"ID":"8af18a66-0fdb-4655-82d9-5716c7e55dae","Type":"ContainerDied","Data":"2f46446bd05cd779350d16d9192a7c7427d041e3a163fe3c6972008fcc310400"} Jul 24 19:53:23.110288 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.110274 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4" Jul 24 19:53:23.110823 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.110320 2570 scope.go:117] "RemoveContainer" containerID="f72e0d8094b8577c7b29851b520e5243724085e3e584b4cd3ab5d79d3e207a8c" Jul 24 19:53:23.127864 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.127833 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4"] Jul 24 19:53:23.134649 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.134622 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9bce955c-openshift-default-c5c585db-746n4"] Jul 24 19:53:23.230845 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.230813 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:23.597575 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.597527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:23.694701 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:23.694668 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:53:24.582384 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:24.582352 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" path="/var/lib/kubelet/pods/8af18a66-0fdb-4655-82d9-5716c7e55dae/volumes" Jul 24 19:53:47.859952 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.859915 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4"] Jul 24 19:53:47.860385 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860369 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" Jul 24 19:53:47.860441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860388 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" Jul 24 19:53:47.860441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860418 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbfd853-1436-42b8-a6b6-e7505989f529" containerName="payload-pre-processing" Jul 24 19:53:47.860441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860424 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbfd853-1436-42b8-a6b6-e7505989f529" containerName="payload-pre-processing" Jul 24 19:53:47.860441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860432 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" Jul 24 19:53:47.860441 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860438 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" Jul 24 19:53:47.860652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860524 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" Jul 24 19:53:47.860652 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.860535 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbfd853-1436-42b8-a6b6-e7505989f529" containerName="payload-pre-processing" Jul 24 19:53:47.863712 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.863696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" Jul 24 19:53:47.866055 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.866027 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqw4p\"" Jul 24 19:53:47.870808 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:47.870777 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4"] Jul 24 19:53:48.030882 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:48.030840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05076782-0043-494f-9eb6-b619b24ba47a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4\" (UID: \"05076782-0043-494f-9eb6-b619b24ba47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" Jul 24 19:53:48.131623 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:48.131518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05076782-0043-494f-9eb6-b619b24ba47a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4\" (UID: \"05076782-0043-494f-9eb6-b619b24ba47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" Jul 24 19:53:48.132208 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:48.132180 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05076782-0043-494f-9eb6-b619b24ba47a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4\" (UID: \"05076782-0043-494f-9eb6-b619b24ba47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" Jul 24 19:53:48.177529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:48.177496 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" Jul 24 19:53:48.314075 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:48.314050 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4"] Jul 24 19:53:48.316740 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:53:48.316715 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05076782_0043_494f_9eb6_b619b24ba47a.slice/crio-bb9f3fbd01d03e7322983cebb5960e42b881eac319ee750de6d4ba8a69da0950 WatchSource:0}: Error finding container bb9f3fbd01d03e7322983cebb5960e42b881eac319ee750de6d4ba8a69da0950: Status 404 returned error can't find the container with id bb9f3fbd01d03e7322983cebb5960e42b881eac319ee750de6d4ba8a69da0950 Jul 24 19:53:48.319052 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:48.319023 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 19:53:49.253677 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:53:49.253636 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" event={"ID":"05076782-0043-494f-9eb6-b619b24ba47a","Type":"ContainerStarted","Data":"bb9f3fbd01d03e7322983cebb5960e42b881eac319ee750de6d4ba8a69da0950"} Jul 24 19:54:11.483786 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:11.483749 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:12.380797 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:12.380759 2570 generic.go:356] "Generic (PLEG): container finished" podID="05076782-0043-494f-9eb6-b619b24ba47a" containerID="0d475c54d63b9ef0230f170f13380002091a929e9a7cd720618a696b372cc8ad" exitCode=0 Jul 24 19:54:12.380979 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:12.380805 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" event={"ID":"05076782-0043-494f-9eb6-b619b24ba47a","Type":"ContainerDied","Data":"0d475c54d63b9ef0230f170f13380002091a929e9a7cd720618a696b372cc8ad"} Jul 24 19:54:13.533934 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:13.533908 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" Jul 24 19:54:13.676548 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:13.676441 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05076782-0043-494f-9eb6-b619b24ba47a-maas-api-service-ca\") pod \"05076782-0043-494f-9eb6-b619b24ba47a\" (UID: \"05076782-0043-494f-9eb6-b619b24ba47a\") " Jul 24 19:54:13.676848 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:13.676820 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05076782-0043-494f-9eb6-b619b24ba47a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05076782-0043-494f-9eb6-b619b24ba47a" (UID: "05076782-0043-494f-9eb6-b619b24ba47a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:54:13.777129 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:13.777088 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05076782-0043-494f-9eb6-b619b24ba47a-maas-api-service-ca\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:54:14.393481 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:14.393448 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" Jul 24 19:54:14.393685 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:14.393485 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4" event={"ID":"05076782-0043-494f-9eb6-b619b24ba47a","Type":"ContainerDied","Data":"bb9f3fbd01d03e7322983cebb5960e42b881eac319ee750de6d4ba8a69da0950"} Jul 24 19:54:14.393685 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:14.393536 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9f3fbd01d03e7322983cebb5960e42b881eac319ee750de6d4ba8a69da0950" Jul 24 19:54:18.312278 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:18.312248 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:18.381065 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:18.381029 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:18.438317 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:18.438284 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:22.946866 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:22.946822 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4"] Jul 24 19:54:22.951191 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:22.951158 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:22.951631 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:22.951612 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-301a8697-ckrg4"] Jul 24 19:54:23.109790 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:23.109760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:23.339216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:23.339178 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:23.570044 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:23.570013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:23.965707 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:23.965673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:24.187422 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:24.187385 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:24.581317 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:24.581283 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05076782-0043-494f-9eb6-b619b24ba47a" path="/var/lib/kubelet/pods/05076782-0043-494f-9eb6-b619b24ba47a/volumes" Jul 24 19:54:26.166316 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:26.166280 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:26.263346 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:26.263314 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:26.426772 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:26.426691 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:26.527077 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:26.527044 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:26.909696 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:26.909667 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:27.008404 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.008366 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:54:27.049831 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.049796 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7"] Jul 24 19:54:27.050293 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.050275 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05076782-0043-494f-9eb6-b619b24ba47a" containerName="revoke-keys" Jul 24 19:54:27.050384 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.050296 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="05076782-0043-494f-9eb6-b619b24ba47a" containerName="revoke-keys" Jul 24 19:54:27.050497 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.050455 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="05076782-0043-494f-9eb6-b619b24ba47a" containerName="revoke-keys" Jul 24 19:54:27.050572 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.050528 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af18a66-0fdb-4655-82d9-5716c7e55dae" containerName="istio-proxy" Jul 24 19:54:27.054976 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.054957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.058510 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.058489 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4babb888-dockercfg-ftl74\"" Jul 24 19:54:27.058638 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.058495 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4babb888\"" Jul 24 19:54:27.066793 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.066771 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7"] Jul 24 19:54:27.197538 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.197443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gc2x\" (UniqueName: \"kubernetes.io/projected/3dcd0ac7-99da-472c-9371-1a9f59e0f629-kube-api-access-8gc2x\") pod \"payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.197538 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.197514 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcd0ac7-99da-472c-9371-1a9f59e0f629-plugins-config-volume\") pod \"payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.298369 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.298336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gc2x\" (UniqueName: \"kubernetes.io/projected/3dcd0ac7-99da-472c-9371-1a9f59e0f629-kube-api-access-8gc2x\") pod \"payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.298545 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.298396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcd0ac7-99da-472c-9371-1a9f59e0f629-plugins-config-volume\") pod \"payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.299131 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.299111 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcd0ac7-99da-472c-9371-1a9f59e0f629-plugins-config-volume\") pod \"payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.306426 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.306403 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gc2x\" (UniqueName: \"kubernetes.io/projected/3dcd0ac7-99da-472c-9371-1a9f59e0f629-kube-api-access-8gc2x\") pod \"payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.365916 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.365887 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:27.724121 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:27.724097 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7"] Jul 24 19:54:27.725973 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:54:27.725944 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcd0ac7_99da_472c_9371_1a9f59e0f629.slice/crio-065dd094f957a1f52c5437111b0ef9504930d0cc2bd674ce6e2f3dbe1ad4ce99 WatchSource:0}: Error finding container 065dd094f957a1f52c5437111b0ef9504930d0cc2bd674ce6e2f3dbe1ad4ce99: Status 404 returned error can't find the container with id 065dd094f957a1f52c5437111b0ef9504930d0cc2bd674ce6e2f3dbe1ad4ce99 Jul 24 19:54:28.469825 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:28.469786 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" event={"ID":"3dcd0ac7-99da-472c-9371-1a9f59e0f629","Type":"ContainerStarted","Data":"44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4"} Jul 24 19:54:28.470227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:28.469845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" event={"ID":"3dcd0ac7-99da-472c-9371-1a9f59e0f629","Type":"ContainerStarted","Data":"065dd094f957a1f52c5437111b0ef9504930d0cc2bd674ce6e2f3dbe1ad4ce99"} Jul 24 19:54:28.470227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:28.469873 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:54:28.485328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:28.485285 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" podStartSLOduration=1.485270898 podStartE2EDuration="1.485270898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:54:28.485067245 +0000 UTC m=+1698.580556456" watchObservedRunningTime="2026-07-24 19:54:28.485270898 +0000 UTC m=+1698.580760156" Jul 24 19:54:39.478911 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:54:39.478833 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:55:02.529786 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.529728 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:02.551931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.551896 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7"] Jul 24 19:55:02.552140 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.552115 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" podUID="3dcd0ac7-99da-472c-9371-1a9f59e0f629" containerName="payload-processing" containerID="cri-o://44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4" gracePeriod=30 Jul 24 19:55:02.629277 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.629245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:02.688121 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.688091 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:02.806654 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.806632 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:55:02.923953 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.923923 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcd0ac7-99da-472c-9371-1a9f59e0f629-plugins-config-volume\") pod \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " Jul 24 19:55:02.924145 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.923980 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gc2x\" (UniqueName: \"kubernetes.io/projected/3dcd0ac7-99da-472c-9371-1a9f59e0f629-kube-api-access-8gc2x\") pod \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\" (UID: \"3dcd0ac7-99da-472c-9371-1a9f59e0f629\") " Jul 24 19:55:02.924301 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.924270 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcd0ac7-99da-472c-9371-1a9f59e0f629-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3dcd0ac7-99da-472c-9371-1a9f59e0f629" (UID: "3dcd0ac7-99da-472c-9371-1a9f59e0f629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:55:02.926188 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:02.926159 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcd0ac7-99da-472c-9371-1a9f59e0f629-kube-api-access-8gc2x" (OuterVolumeSpecName: "kube-api-access-8gc2x") pod "3dcd0ac7-99da-472c-9371-1a9f59e0f629" (UID: "3dcd0ac7-99da-472c-9371-1a9f59e0f629"). InnerVolumeSpecName "kube-api-access-8gc2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:55:03.024772 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.024744 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcd0ac7-99da-472c-9371-1a9f59e0f629-plugins-config-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:55:03.024772 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.024768 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gc2x\" (UniqueName: \"kubernetes.io/projected/3dcd0ac7-99da-472c-9371-1a9f59e0f629-kube-api-access-8gc2x\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:55:03.654313 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.654280 2570 generic.go:356] "Generic (PLEG): container finished" podID="3dcd0ac7-99da-472c-9371-1a9f59e0f629" containerID="44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4" exitCode=0 Jul 24 19:55:03.654759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.654338 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" Jul 24 19:55:03.654759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.654365 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" event={"ID":"3dcd0ac7-99da-472c-9371-1a9f59e0f629","Type":"ContainerDied","Data":"44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4"} Jul 24 19:55:03.654759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.654419 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7" event={"ID":"3dcd0ac7-99da-472c-9371-1a9f59e0f629","Type":"ContainerDied","Data":"065dd094f957a1f52c5437111b0ef9504930d0cc2bd674ce6e2f3dbe1ad4ce99"} Jul 24 19:55:03.654759 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.654441 2570 scope.go:117] "RemoveContainer" containerID="44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4" Jul 24 19:55:03.664109 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.664092 2570 scope.go:117] "RemoveContainer" containerID="44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4" Jul 24 19:55:03.664360 ip-10-0-141-238 kubenswrapper[2570]: E0724 19:55:03.664338 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4\": container with ID starting with 44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4 not found: ID does not exist" containerID="44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4" Jul 24 19:55:03.664422 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.664372 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4"} err="failed to get container status \"44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4\": rpc error: code = NotFound desc = could not find container \"44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4\": container with ID starting with 44375fc7883c083c51b8d8160790919245cbbbf4476b90e220a49054e488d2e4 not found: ID does not exist" Jul 24 19:55:03.674856 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.674835 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7"] Jul 24 19:55:03.679001 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:03.678980 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4babb888-56d44bbf7d-ptgf7"] Jul 24 19:55:04.581669 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:04.581629 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dcd0ac7-99da-472c-9371-1a9f59e0f629" path="/var/lib/kubelet/pods/3dcd0ac7-99da-472c-9371-1a9f59e0f629/volumes" Jul 24 19:55:06.941841 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:06.941810 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:08.110108 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:08.110077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:08.342830 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:08.342797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:08.550815 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:08.550782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:08.869016 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:08.868938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:09.383405 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:09.383370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:11.413721 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:11.413692 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:11.515225 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:11.515183 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:11.639295 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:11.639264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:12.006398 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.006359 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:12.169713 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.169669 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:12.269479 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.269386 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:12.304324 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.304290 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4"] Jul 24 19:55:12.304891 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.304869 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dcd0ac7-99da-472c-9371-1a9f59e0f629" containerName="payload-processing" Jul 24 19:55:12.304891 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.304891 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dcd0ac7-99da-472c-9371-1a9f59e0f629" containerName="payload-processing" Jul 24 19:55:12.305100 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.304986 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dcd0ac7-99da-472c-9371-1a9f59e0f629" containerName="payload-processing" Jul 24 19:55:12.309572 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.309547 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.312691 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.312671 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9b3176cc-dockercfg-vzpg8\"" Jul 24 19:55:12.313426 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.313105 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9b3176cc\"" Jul 24 19:55:12.317098 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.317075 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4"] Jul 24 19:55:12.412907 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.412870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036e04d9-c3c7-4eca-9e7d-b57777d960e9-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.413103 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.412932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnkc\" (UniqueName: \"kubernetes.io/projected/036e04d9-c3c7-4eca-9e7d-b57777d960e9-kube-api-access-jwnkc\") pod \"payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.514160 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.514124 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036e04d9-c3c7-4eca-9e7d-b57777d960e9-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.514560 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.514168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnkc\" (UniqueName: \"kubernetes.io/projected/036e04d9-c3c7-4eca-9e7d-b57777d960e9-kube-api-access-jwnkc\") pod \"payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.514881 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.514850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036e04d9-c3c7-4eca-9e7d-b57777d960e9-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.523072 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.523021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnkc\" (UniqueName: \"kubernetes.io/projected/036e04d9-c3c7-4eca-9e7d-b57777d960e9-kube-api-access-jwnkc\") pod \"payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.624553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.624524 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:12.973710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:12.973502 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4"] Jul 24 19:55:12.976664 ip-10-0-141-238 kubenswrapper[2570]: W0724 19:55:12.976634 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036e04d9_c3c7_4eca_9e7d_b57777d960e9.slice/crio-e0dd14b06ddfc63003b70f1aeb1b062c735a87495c2bdacefecdcebbfde040a2 WatchSource:0}: Error finding container e0dd14b06ddfc63003b70f1aeb1b062c735a87495c2bdacefecdcebbfde040a2: Status 404 returned error can't find the container with id e0dd14b06ddfc63003b70f1aeb1b062c735a87495c2bdacefecdcebbfde040a2 Jul 24 19:55:13.710931 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:13.710895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" event={"ID":"036e04d9-c3c7-4eca-9e7d-b57777d960e9","Type":"ContainerStarted","Data":"b6df2f25d745f8c7fa80d784b4969974e0a4b17511cffe396f3c773cd0491b45"} Jul 24 19:55:13.711328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:13.710950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" event={"ID":"036e04d9-c3c7-4eca-9e7d-b57777d960e9","Type":"ContainerStarted","Data":"e0dd14b06ddfc63003b70f1aeb1b062c735a87495c2bdacefecdcebbfde040a2"} Jul 24 19:55:13.711328 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:13.711036 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:13.730379 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:13.730333 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" podStartSLOduration=1.730319686 podStartE2EDuration="1.730319686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:55:13.728612466 +0000 UTC m=+1743.824101673" watchObservedRunningTime="2026-07-24 19:55:13.730319686 +0000 UTC m=+1743.825808893" Jul 24 19:55:24.718029 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:24.717995 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:46.729653 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:46.729615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:46.754381 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:46.754346 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4"] Jul 24 19:55:46.754820 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:46.754790 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" podUID="036e04d9-c3c7-4eca-9e7d-b57777d960e9" containerName="payload-processing" containerID="cri-o://b6df2f25d745f8c7fa80d784b4969974e0a4b17511cffe396f3c773cd0491b45" gracePeriod=30 Jul 24 19:55:46.828433 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:46.828403 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:46.879594 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:46.879566 2570 generic.go:356] "Generic (PLEG): container finished" podID="036e04d9-c3c7-4eca-9e7d-b57777d960e9" containerID="b6df2f25d745f8c7fa80d784b4969974e0a4b17511cffe396f3c773cd0491b45" exitCode=0 Jul 24 19:55:46.879710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:46.879634 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" event={"ID":"036e04d9-c3c7-4eca-9e7d-b57777d960e9","Type":"ContainerDied","Data":"b6df2f25d745f8c7fa80d784b4969974e0a4b17511cffe396f3c773cd0491b45"} Jul 24 19:55:46.930803 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:46.930715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:47.009612 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.009589 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:47.124590 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.124561 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036e04d9-c3c7-4eca-9e7d-b57777d960e9-plugins-config-volume\") pod \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " Jul 24 19:55:47.124751 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.124597 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwnkc\" (UniqueName: \"kubernetes.io/projected/036e04d9-c3c7-4eca-9e7d-b57777d960e9-kube-api-access-jwnkc\") pod \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\" (UID: \"036e04d9-c3c7-4eca-9e7d-b57777d960e9\") " Jul 24 19:55:47.124996 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.124971 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036e04d9-c3c7-4eca-9e7d-b57777d960e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "036e04d9-c3c7-4eca-9e7d-b57777d960e9" (UID: "036e04d9-c3c7-4eca-9e7d-b57777d960e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:55:47.126649 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.126627 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036e04d9-c3c7-4eca-9e7d-b57777d960e9-kube-api-access-jwnkc" (OuterVolumeSpecName: "kube-api-access-jwnkc") pod "036e04d9-c3c7-4eca-9e7d-b57777d960e9" (UID: "036e04d9-c3c7-4eca-9e7d-b57777d960e9"). InnerVolumeSpecName "kube-api-access-jwnkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:55:47.226180 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.226152 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036e04d9-c3c7-4eca-9e7d-b57777d960e9-plugins-config-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:55:47.226180 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.226176 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnkc\" (UniqueName: \"kubernetes.io/projected/036e04d9-c3c7-4eca-9e7d-b57777d960e9-kube-api-access-jwnkc\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 19:55:47.886404 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.886365 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" event={"ID":"036e04d9-c3c7-4eca-9e7d-b57777d960e9","Type":"ContainerDied","Data":"e0dd14b06ddfc63003b70f1aeb1b062c735a87495c2bdacefecdcebbfde040a2"} Jul 24 19:55:47.886816 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.886433 2570 scope.go:117] "RemoveContainer" containerID="b6df2f25d745f8c7fa80d784b4969974e0a4b17511cffe396f3c773cd0491b45" Jul 24 19:55:47.886816 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.886382 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4" Jul 24 19:55:47.909592 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.909564 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4"] Jul 24 19:55:47.913422 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:47.913400 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b3176cc-7f4f66fc66-z69t4"] Jul 24 19:55:48.582262 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:48.582225 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="036e04d9-c3c7-4eca-9e7d-b57777d960e9" path="/var/lib/kubelet/pods/036e04d9-c3c7-4eca-9e7d-b57777d960e9/volumes" Jul 24 19:55:51.290153 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:51.290105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:51.444692 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:51.444659 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:51.685667 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:51.685591 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:52.230614 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:52.230570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:52.408205 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:52.408177 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:53.846058 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:53.846025 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:55:59.946990 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:55:59.946957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:00.052529 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:00.052496 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:00.149413 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:00.149384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:00.260162 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:00.260132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:00.718952 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:00.718858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:02.540777 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:02.540742 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:02.610100 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:02.610071 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:10.497646 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:10.497619 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:56:10.501079 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:10.501057 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 19:56:26.648742 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:26.648707 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:33.311888 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:33.311850 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:33.370989 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:33.370956 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:33.443054 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:33.443019 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:33.540924 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:33.540889 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:49.402282 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:49.402245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:50.687406 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:50.687371 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:50.736772 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:50.736741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:55.462111 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:55.462069 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:55.520777 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:55.520740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:55.654257 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:55.654228 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:56:55.712424 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:56:55.712354 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:06.152833 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:06.152794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:06.980854 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:06.980825 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:14.050723 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:14.050681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:14.163108 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:14.163078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:14.252584 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:14.252556 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:14.354844 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:14.354748 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:15.154476 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:15.154425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:15.247216 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:15.247178 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:15.345436 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:15.345402 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:16.992113 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:16.992080 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:22.813528 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:22.813461 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:22.874730 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:22.874698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:23.476925 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:23.476892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:29.873381 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:29.873351 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:29.972777 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:29.972747 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:30.104792 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:30.104743 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:30.424586 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:30.424554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:30.615668 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:30.615633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:33.607666 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:33.607579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:33.696691 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:33.696656 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:33.775858 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:33.775823 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:33.983423 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:33.983326 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:34.254481 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:34.254443 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:34.350944 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:34.350910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:38.106593 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:38.106562 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:38.767184 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:38.767147 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:38.866710 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:38.866680 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:39.288307 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:39.288276 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:42.354937 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:42.354896 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:42.452801 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:42.452763 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:45.321653 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:45.321617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:45.421839 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:45.421808 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:55.803836 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:55.803790 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:55.867084 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:55.867051 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:57:55.968633 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:57:55.968601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:00.748906 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:00.748870 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:01.139171 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:01.139086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:06.736520 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:06.736487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:06.836723 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:06.836693 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:07.024175 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:07.024094 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:07.242397 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:07.242363 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:07.402345 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:07.402257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:07.574074 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:07.574039 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:07.672559 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:07.672481 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:13.427335 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:13.427303 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:13.528861 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:13.528828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:13.624052 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:13.624022 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:14.746505 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:14.746458 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:19.751553 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:19.751459 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:19.852031 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:19.851995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:26.557537 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:26.557494 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:26.857671 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:26.857572 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:27.026134 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:27.026096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:29.846559 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:29.846527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:29.946967 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:29.946933 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:30.127219 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:30.127131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:30.403227 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:30.403135 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:30.623195 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:30.623163 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:30.725574 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:30.725443 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:58:34.364981 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:58:34.364942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:59:00.685785 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:59:00.685751 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:59:32.040197 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:59:32.040159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:59:44.670809 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:59:44.670772 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:59:44.766342 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:59:44.766306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:59:49.271936 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:59:49.271901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:59:52.484750 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:59:52.484713 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 19:59:57.497689 ip-10-0-141-238 kubenswrapper[2570]: I0724 19:59:57.497651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:17.240262 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:17.240227 2570 scope.go:117] "RemoveContainer" containerID="0d475c54d63b9ef0230f170f13380002091a929e9a7cd720618a696b372cc8ad" Jul 24 20:00:34.252700 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:34.252619 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:34.676621 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:34.676539 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:40.164978 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:40.164942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:40.225454 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:40.225421 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:40.454901 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:40.454811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:40.684073 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:40.684035 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:41.758576 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:41.758541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:41.857397 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:41.857361 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:41.963299 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:41.963263 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:42.685347 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:42.685310 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:42.801366 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:42.801333 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:42.972851 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:42.972766 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:44.289679 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:44.289647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:49.866173 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:49.866132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:49.966458 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:49.966422 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:50.539428 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:50.539395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:56.684925 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:56.684869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:56.794120 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:56.794081 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:00:56.887862 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:00:56.887827 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:07.413439 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:07.413406 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:07.777343 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:07.777315 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:10.536624 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:10.536594 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:01:10.544839 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:10.544814 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:01:28.761784 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:28.761754 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:34.993479 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:34.993421 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:35.092974 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:35.092940 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:35.356061 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:35.355967 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:35.670077 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:35.669995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:35.765931 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:35.765897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:38.842049 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:38.842004 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:39.002310 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:39.002270 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:39.095234 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:39.095152 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:39.742940 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:39.742901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:39.839921 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:39.839887 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:43.294647 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.294607 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:43.634804 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.634720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:43.913875 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.913781 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9"] Jul 24 20:01:43.914422 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.914401 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="036e04d9-c3c7-4eca-9e7d-b57777d960e9" containerName="payload-processing" Jul 24 20:01:43.914422 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.914420 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="036e04d9-c3c7-4eca-9e7d-b57777d960e9" containerName="payload-processing" Jul 24 20:01:43.914601 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.914550 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="036e04d9-c3c7-4eca-9e7d-b57777d960e9" containerName="payload-processing" Jul 24 20:01:43.918173 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.918155 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:43.920637 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.920618 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6a7079-openshift-default-dockercfg-dm4fz\"" Jul 24 20:01:43.929401 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.929381 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9"] Jul 24 20:01:43.936825 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:43.936801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:44.021965 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.021922 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-data\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022162 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.021975 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-socket\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022162 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.022085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrt45\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-kube-api-access-vrt45\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.022169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-envoy\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.022224 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-certs\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.022248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-podinfo\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.022274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a706e8-43b3-4363-ad74-2f032de3cd02-istiod-ca-cert\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022544 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.022322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-credential-socket\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.022544 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.022363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-token\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.092392 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.092359 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:44.123877 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.123843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-token\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124035 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.123884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-data\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124035 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.123906 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-socket\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124035 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.123947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt45\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-kube-api-access-vrt45\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124035 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.123986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-envoy\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124217 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124039 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-certs\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124217 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124068 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-podinfo\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124217 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a706e8-43b3-4363-ad74-2f032de3cd02-istiod-ca-cert\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124217 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-credential-socket\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124434 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-data\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124434 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124394 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-socket\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124608 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124587 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-certs\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124663 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124639 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-credential-socket\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.124845 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.124824 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a706e8-43b3-4363-ad74-2f032de3cd02-istiod-ca-cert\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.126352 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.126319 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-envoy\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.126658 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.126641 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-podinfo\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.131434 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.131413 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-token\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.131588 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.131528 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt45\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-kube-api-access-vrt45\") pod \"e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.229993 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.229905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:44.371134 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.371111 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9"] Jul 24 20:01:44.372803 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:01:44.372769 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a706e8_43b3_4363_ad74_2f032de3cd02.slice/crio-dd8525f144317f2f578836473e80c3ec3554d7c89f390228daf80e94940fd4a8 WatchSource:0}: Error finding container dd8525f144317f2f578836473e80c3ec3554d7c89f390228daf80e94940fd4a8: Status 404 returned error can't find the container with id dd8525f144317f2f578836473e80c3ec3554d7c89f390228daf80e94940fd4a8 Jul 24 20:01:44.375044 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.375005 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:01:44.375124 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.375097 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:01:44.375166 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.375144 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:01:44.782318 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.782278 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" event={"ID":"f6a706e8-43b3-4363-ad74-2f032de3cd02","Type":"ContainerStarted","Data":"169b6fd50ad1a2696b1f36ed0cb99b79240d0fd65b35457f68889330976a1ff2"} Jul 24 20:01:44.782538 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.782343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" event={"ID":"f6a706e8-43b3-4363-ad74-2f032de3cd02","Type":"ContainerStarted","Data":"dd8525f144317f2f578836473e80c3ec3554d7c89f390228daf80e94940fd4a8"} Jul 24 20:01:44.801653 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:44.801603 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" podStartSLOduration=1.801586319 podStartE2EDuration="1.801586319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:01:44.80034324 +0000 UTC m=+2134.895832444" watchObservedRunningTime="2026-07-24 20:01:44.801586319 +0000 UTC m=+2134.897075526" Jul 24 20:01:45.231039 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:45.230957 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:45.236746 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:45.236718 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:45.787599 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:45.787568 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:45.788616 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:45.788588 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:01:46.744330 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:46.744294 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:46.848030 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:46.847998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:46.922583 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:46.922543 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:47.410329 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.410296 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:47.567900 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.567857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:47.642737 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.642704 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:47.766064 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.766032 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f"] Jul 24 20:01:47.769813 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.769785 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:47.772135 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.772072 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6a7079\"" Jul 24 20:01:47.772135 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.772100 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6a7079-dockercfg-v4crd\"" Jul 24 20:01:47.778866 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.778533 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f"] Jul 24 20:01:47.866105 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.866059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b82707-af30-419f-8e43-cd3dd625c17f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:47.866628 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.866112 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqss\" (UniqueName: \"kubernetes.io/projected/32b82707-af30-419f-8e43-cd3dd625c17f-kube-api-access-blqss\") pod \"payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:47.967755 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.967719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b82707-af30-419f-8e43-cd3dd625c17f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:47.967755 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.967760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqss\" (UniqueName: \"kubernetes.io/projected/32b82707-af30-419f-8e43-cd3dd625c17f-kube-api-access-blqss\") pod \"payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:47.968548 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.968522 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b82707-af30-419f-8e43-cd3dd625c17f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:47.975661 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:47.975640 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqss\" (UniqueName: \"kubernetes.io/projected/32b82707-af30-419f-8e43-cd3dd625c17f-kube-api-access-blqss\") pod \"payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:48.083001 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:48.082914 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:48.427507 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:48.426865 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f"] Jul 24 20:01:48.428266 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:01:48.428240 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b82707_af30_419f_8e43_cd3dd625c17f.slice/crio-a2332fa52a20cba8ac163a3e71b46e051eec2bb3683054993dfa6ce9e2cd298a WatchSource:0}: Error finding container a2332fa52a20cba8ac163a3e71b46e051eec2bb3683054993dfa6ce9e2cd298a: Status 404 returned error can't find the container with id a2332fa52a20cba8ac163a3e71b46e051eec2bb3683054993dfa6ce9e2cd298a Jul 24 20:01:48.804536 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:48.804503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" event={"ID":"32b82707-af30-419f-8e43-cd3dd625c17f","Type":"ContainerStarted","Data":"84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc"} Jul 24 20:01:48.804716 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:48.804552 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" event={"ID":"32b82707-af30-419f-8e43-cd3dd625c17f","Type":"ContainerStarted","Data":"a2332fa52a20cba8ac163a3e71b46e051eec2bb3683054993dfa6ce9e2cd298a"} Jul 24 20:01:48.804716 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:48.804633 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:48.820719 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:48.820667 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" podStartSLOduration=1.820652256 podStartE2EDuration="1.820652256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:01:48.819032648 +0000 UTC m=+2138.914521856" watchObservedRunningTime="2026-07-24 20:01:48.820652256 +0000 UTC m=+2138.916141463" Jul 24 20:01:51.241296 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:51.241258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:56.037996 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:56.037959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:01:59.811943 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:59.811908 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:01:59.873272 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:59.873187 2570 generic.go:356] "Generic (PLEG): container finished" podID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerID="169b6fd50ad1a2696b1f36ed0cb99b79240d0fd65b35457f68889330976a1ff2" exitCode=0 Jul 24 20:01:59.873427 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:59.873260 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" event={"ID":"f6a706e8-43b3-4363-ad74-2f032de3cd02","Type":"ContainerDied","Data":"169b6fd50ad1a2696b1f36ed0cb99b79240d0fd65b35457f68889330976a1ff2"} Jul 24 20:01:59.873716 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:59.873702 2570 scope.go:117] "RemoveContainer" containerID="169b6fd50ad1a2696b1f36ed0cb99b79240d0fd65b35457f68889330976a1ff2" Jul 24 20:01:59.874828 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:59.874795 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:01:59.874937 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:59.874888 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:01:59.874998 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:01:59.874935 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:02:00.788317 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:00.788284 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:02:00.881556 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:00.881516 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" event={"ID":"f6a706e8-43b3-4363-ad74-2f032de3cd02","Type":"ContainerStarted","Data":"2b3c0affb1fbc93e67019518b3907be88b7fbb7762ce18d471f87eaf6f259f06"} Jul 24 20:02:01.230553 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:01.230453 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:02:01.231904 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:01.231865 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 20:02:01.232049 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:01.231931 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 20:02:02.230297 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:02.230262 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 20:02:02.230753 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:02.230324 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 20:02:03.230589 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:03.230502 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 20:02:03.230589 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:03.230557 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 20:02:04.230365 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:04.230331 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 20:02:04.230565 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:04.230384 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 20:02:05.234561 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:05.234529 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:02:05.235039 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:05.234920 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:02:05.235613 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:05.235594 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:02:11.627284 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:11.627253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:02:50.349593 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:50.349560 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:02:51.311276 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:51.311236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:02:56.618527 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:02:56.618489 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:01.878878 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:01.878834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:01.980576 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:01.980541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:06.543170 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:06.543134 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:06.907691 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:06.907613 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:13.013252 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:13.013217 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:13.071699 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:13.071667 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:13.299519 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:13.299413 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:13.543900 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:13.543869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:24.208399 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.208370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:24.239992 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.239963 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f"] Jul 24 20:03:24.240179 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.240155 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" podUID="32b82707-af30-419f-8e43-cd3dd625c17f" containerName="payload-pre-processing" containerID="cri-o://84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc" gracePeriod=30 Jul 24 20:03:24.510574 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.510544 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:03:24.608540 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.608511 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blqss\" (UniqueName: \"kubernetes.io/projected/32b82707-af30-419f-8e43-cd3dd625c17f-kube-api-access-blqss\") pod \"32b82707-af30-419f-8e43-cd3dd625c17f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " Jul 24 20:03:24.608719 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.608581 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b82707-af30-419f-8e43-cd3dd625c17f-plugins-config-volume\") pod \"32b82707-af30-419f-8e43-cd3dd625c17f\" (UID: \"32b82707-af30-419f-8e43-cd3dd625c17f\") " Jul 24 20:03:24.608962 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.608939 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b82707-af30-419f-8e43-cd3dd625c17f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32b82707-af30-419f-8e43-cd3dd625c17f" (UID: "32b82707-af30-419f-8e43-cd3dd625c17f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 20:03:24.610572 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.610548 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b82707-af30-419f-8e43-cd3dd625c17f-kube-api-access-blqss" (OuterVolumeSpecName: "kube-api-access-blqss") pod "32b82707-af30-419f-8e43-cd3dd625c17f" (UID: "32b82707-af30-419f-8e43-cd3dd625c17f"). InnerVolumeSpecName "kube-api-access-blqss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:03:24.709909 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.709868 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqss\" (UniqueName: \"kubernetes.io/projected/32b82707-af30-419f-8e43-cd3dd625c17f-kube-api-access-blqss\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:24.709909 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:24.709902 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b82707-af30-419f-8e43-cd3dd625c17f-plugins-config-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:25.333887 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.333856 2570 generic.go:356] "Generic (PLEG): container finished" podID="32b82707-af30-419f-8e43-cd3dd625c17f" containerID="84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc" exitCode=0 Jul 24 20:03:25.334292 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.333898 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" Jul 24 20:03:25.334292 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.333892 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" event={"ID":"32b82707-af30-419f-8e43-cd3dd625c17f","Type":"ContainerDied","Data":"84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc"} Jul 24 20:03:25.334292 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.334016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f" event={"ID":"32b82707-af30-419f-8e43-cd3dd625c17f","Type":"ContainerDied","Data":"a2332fa52a20cba8ac163a3e71b46e051eec2bb3683054993dfa6ce9e2cd298a"} Jul 24 20:03:25.334292 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.334040 2570 scope.go:117] "RemoveContainer" containerID="84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc" Jul 24 20:03:25.344149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.344125 2570 scope.go:117] "RemoveContainer" containerID="84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc" Jul 24 20:03:25.344430 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:03:25.344411 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc\": container with ID starting with 84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc not found: ID does not exist" containerID="84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc" Jul 24 20:03:25.344503 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.344439 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc"} err="failed to get container status \"84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc\": rpc error: code = NotFound desc = could not find container \"84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc\": container with ID starting with 84dbb18bf4b87c23a6388ab1d7bac5f4b66d7e8b6c68befeb4da6e0e34da01dc not found: ID does not exist" Jul 24 20:03:25.356132 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.356110 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f"] Jul 24 20:03:25.359526 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.359507 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6a7079-649b7bdcc6-x596f"] Jul 24 20:03:25.429437 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:25.429412 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:26.582007 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:26.581973 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32b82707-af30-419f-8e43-cd3dd625c17f" path="/var/lib/kubelet/pods/32b82707-af30-419f-8e43-cd3dd625c17f/volumes" Jul 24 20:03:28.810348 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:28.810314 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:29.214877 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:29.214800 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:34.979163 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:34.979083 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:35.049701 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.049668 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:35.233822 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.233748 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9"] Jul 24 20:03:35.233979 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.233943 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" containerID="cri-o://2b3c0affb1fbc93e67019518b3907be88b7fbb7762ce18d471f87eaf6f259f06" gracePeriod=30 Jul 24 20:03:35.237615 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.237581 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 20:03:35.237756 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.237635 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 20:03:35.248171 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.248151 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:35.476045 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.476005 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:35.815287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:35.815256 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:36.133649 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:36.133558 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:36.754527 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:36.754497 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:39.377765 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:39.377720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:39.474098 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:39.474062 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:39.550129 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:39.550092 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:39.731677 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:39.731591 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:40.128256 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.128229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:40.233436 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.233397 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:40.426154 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.426122 2570 generic.go:356] "Generic (PLEG): container finished" podID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerID="2b3c0affb1fbc93e67019518b3907be88b7fbb7762ce18d471f87eaf6f259f06" exitCode=0 Jul 24 20:03:40.426551 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.426177 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" event={"ID":"f6a706e8-43b3-4363-ad74-2f032de3cd02","Type":"ContainerDied","Data":"2b3c0affb1fbc93e67019518b3907be88b7fbb7762ce18d471f87eaf6f259f06"} Jul 24 20:03:40.426551 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.426236 2570 scope.go:117] "RemoveContainer" containerID="169b6fd50ad1a2696b1f36ed0cb99b79240d0fd65b35457f68889330976a1ff2" Jul 24 20:03:40.506377 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.506352 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:03:40.553662 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.553634 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-token\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.553787 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.553673 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-credential-socket\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.553787 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.553710 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-data\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.553894 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.553853 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a706e8-43b3-4363-ad74-2f032de3cd02-istiod-ca-cert\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.554006 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.553978 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:03:40.554075 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.553994 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrt45\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-kube-api-access-vrt45\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.554133 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554089 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-socket\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.554133 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554111 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-podinfo\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.554241 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554158 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-certs\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.554241 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554175 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a706e8-43b3-4363-ad74-2f032de3cd02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 20:03:40.554241 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554188 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-envoy\") pod \"f6a706e8-43b3-4363-ad74-2f032de3cd02\" (UID: \"f6a706e8-43b3-4363-ad74-2f032de3cd02\") " Jul 24 20:03:40.554642 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554450 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:03:40.554642 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554538 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:03:40.554642 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554573 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-certs\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.554642 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554592 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-credential-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.554642 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554609 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a706e8-43b3-4363-ad74-2f032de3cd02-istiod-ca-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.555113 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.554786 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:03:40.556147 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.556117 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-kube-api-access-vrt45" (OuterVolumeSpecName: "kube-api-access-vrt45") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "kube-api-access-vrt45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:03:40.556251 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.556235 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 20:03:40.556333 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.556318 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:03:40.556626 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.556612 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6a706e8-43b3-4363-ad74-2f032de3cd02" (UID: "f6a706e8-43b3-4363-ad74-2f032de3cd02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:03:40.655741 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.655697 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-token\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.655741 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.655729 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-data\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.655741 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.655742 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrt45\" (UniqueName: \"kubernetes.io/projected/f6a706e8-43b3-4363-ad74-2f032de3cd02-kube-api-access-vrt45\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.655986 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.655760 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-workload-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.655986 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.655774 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-podinfo\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.655986 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.655787 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a706e8-43b3-4363-ad74-2f032de3cd02-istio-envoy\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:03:40.937328 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:40.937298 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:41.040705 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:41.040672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:41.432841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:41.432801 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" Jul 24 20:03:41.432841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:41.432839 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9" event={"ID":"f6a706e8-43b3-4363-ad74-2f032de3cd02","Type":"ContainerDied","Data":"dd8525f144317f2f578836473e80c3ec3554d7c89f390228daf80e94940fd4a8"} Jul 24 20:03:41.433334 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:41.432902 2570 scope.go:117] "RemoveContainer" containerID="2b3c0affb1fbc93e67019518b3907be88b7fbb7762ce18d471f87eaf6f259f06" Jul 24 20:03:41.450688 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:41.450660 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9"] Jul 24 20:03:41.454248 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:41.454226 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6a7079-openshift-default-787bbc8b44-ckjg9"] Jul 24 20:03:42.581704 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:42.581671 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" path="/var/lib/kubelet/pods/f6a706e8-43b3-4363-ad74-2f032de3cd02/volumes" Jul 24 20:03:43.880430 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:43.880390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:44.322837 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.322771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:44.615700 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.615618 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx"] Jul 24 20:03:44.616258 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616241 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" Jul 24 20:03:44.616314 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616262 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" Jul 24 20:03:44.616314 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616277 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" Jul 24 20:03:44.616314 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616311 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" Jul 24 20:03:44.616411 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616324 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b82707-af30-419f-8e43-cd3dd625c17f" containerName="payload-pre-processing" Jul 24 20:03:44.616411 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616333 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b82707-af30-419f-8e43-cd3dd625c17f" containerName="payload-pre-processing" Jul 24 20:03:44.616492 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616430 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" Jul 24 20:03:44.616492 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616442 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a706e8-43b3-4363-ad74-2f032de3cd02" containerName="istio-proxy" Jul 24 20:03:44.616492 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.616455 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b82707-af30-419f-8e43-cd3dd625c17f" containerName="payload-pre-processing" Jul 24 20:03:44.621057 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.621033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.624148 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.624127 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e316a0-openshift-default-dockercfg-wdhwt\"" Jul 24 20:03:44.632325 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.632293 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx"] Jul 24 20:03:44.652734 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.652707 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:44.689008 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.688985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-podinfo\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689137 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689028 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-credential-socket\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689137 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-data\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689261 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689138 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istiod-ca-cert\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689261 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689197 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-envoy\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689261 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-token\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689384 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-certs\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689384 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzl9s\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-kube-api-access-fzl9s\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.689384 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.689302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-socket\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790133 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-data\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790318 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istiod-ca-cert\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790318 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-envoy\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790481 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-token\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790559 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790506 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-data\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790559 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-certs\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790670 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzl9s\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-kube-api-access-fzl9s\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790670 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790596 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-socket\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790781 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-podinfo\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790781 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-credential-socket\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.790906 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.790882 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-certs\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.791027 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.791003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-socket\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.791085 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.791061 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istiod-ca-cert\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.791132 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.791093 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-credential-socket\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.792629 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.792611 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-envoy\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.793037 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.792999 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-podinfo\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.798157 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.798128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzl9s\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-kube-api-access-fzl9s\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.798643 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.798625 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-token\") pod \"e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:44.934627 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:44.934542 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:45.019491 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.019447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:45.078600 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.078572 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx"] Jul 24 20:03:45.080575 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:03:45.080543 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5e1a44_fdf7_4a6c_8388_2c49d786a992.slice/crio-23f1f17dafded9f32ac4690a35ef08a0d1cbd5113a9ae23bdc518baff90cab28 WatchSource:0}: Error finding container 23f1f17dafded9f32ac4690a35ef08a0d1cbd5113a9ae23bdc518baff90cab28: Status 404 returned error can't find the container with id 23f1f17dafded9f32ac4690a35ef08a0d1cbd5113a9ae23bdc518baff90cab28 Jul 24 20:03:45.083146 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.083106 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:03:45.083242 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.083199 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:03:45.083287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.083244 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:03:45.455478 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.455424 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" event={"ID":"3f5e1a44-fdf7-4a6c-8388-2c49d786a992","Type":"ContainerStarted","Data":"f644c3d16fda1d61f98310b27bae34516d3ebcd7365ec66cd3a60a1b9e33fe4c"} Jul 24 20:03:45.455651 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.455504 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" event={"ID":"3f5e1a44-fdf7-4a6c-8388-2c49d786a992","Type":"ContainerStarted","Data":"23f1f17dafded9f32ac4690a35ef08a0d1cbd5113a9ae23bdc518baff90cab28"} Jul 24 20:03:45.475381 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.475333 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podStartSLOduration=1.4753175889999999 podStartE2EDuration="1.475317589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:03:45.473720849 +0000 UTC m=+2255.569210056" watchObservedRunningTime="2026-07-24 20:03:45.475317589 +0000 UTC m=+2255.570806795" Jul 24 20:03:45.935067 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:45.935032 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:46.940977 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:46.940877 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:47.384160 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:47.384123 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:47.466430 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:47.466399 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:47.467263 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:47.467240 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:03:47.479818 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:47.479791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:47.650045 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:47.649951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:47.998189 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:47.998159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:48.248974 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.248889 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:48.341237 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.341197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:03:48.380057 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.380018 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x"] Jul 24 20:03:48.383762 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.383740 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.386185 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.386167 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e316a0\"" Jul 24 20:03:48.386268 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.386167 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e316a0-dockercfg-b9b8h\"" Jul 24 20:03:48.397291 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.397261 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x"] Jul 24 20:03:48.428701 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.428676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkj7w\" (UniqueName: \"kubernetes.io/projected/2a8b1631-1cca-47e4-a5cd-0cd855a16997-kube-api-access-wkj7w\") pod \"payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.428813 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.428730 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8b1631-1cca-47e4-a5cd-0cd855a16997-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.530162 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.530062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8b1631-1cca-47e4-a5cd-0cd855a16997-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.530364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.530339 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkj7w\" (UniqueName: \"kubernetes.io/projected/2a8b1631-1cca-47e4-a5cd-0cd855a16997-kube-api-access-wkj7w\") pod \"payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.530975 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.530954 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8b1631-1cca-47e4-a5cd-0cd855a16997-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.538716 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.538694 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkj7w\" (UniqueName: \"kubernetes.io/projected/2a8b1631-1cca-47e4-a5cd-0cd855a16997-kube-api-access-wkj7w\") pod \"payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.699810 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.699771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:48.840755 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:48.840724 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x"] Jul 24 20:03:48.842039 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:03:48.842011 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8b1631_1cca_47e4_a5cd_0cd855a16997.slice/crio-30ad650dcfb2eaf23cd32e2a7b62233a3e6b6a155532dac2b000d19a421b31c4 WatchSource:0}: Error finding container 30ad650dcfb2eaf23cd32e2a7b62233a3e6b6a155532dac2b000d19a421b31c4: Status 404 returned error can't find the container with id 30ad650dcfb2eaf23cd32e2a7b62233a3e6b6a155532dac2b000d19a421b31c4 Jul 24 20:03:49.481666 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:49.481625 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" event={"ID":"2a8b1631-1cca-47e4-a5cd-0cd855a16997","Type":"ContainerStarted","Data":"3b2a77068d06aa182b3cfd79ab49065252ac47b0a8849d6d0e861bd17c090d0b"} Jul 24 20:03:49.482086 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:49.481692 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" event={"ID":"2a8b1631-1cca-47e4-a5cd-0cd855a16997","Type":"ContainerStarted","Data":"30ad650dcfb2eaf23cd32e2a7b62233a3e6b6a155532dac2b000d19a421b31c4"} Jul 24 20:03:49.499608 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:49.499561 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" podStartSLOduration=1.49954573 podStartE2EDuration="1.49954573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:03:49.498712317 +0000 UTC m=+2259.594201528" watchObservedRunningTime="2026-07-24 20:03:49.49954573 +0000 UTC m=+2259.595034970" Jul 24 20:03:50.486732 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:50.486701 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:03:51.887294 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:03:51.887257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:01.493839 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:01.493801 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:04:03.507631 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:03.507593 2570 patch_prober.go:28] interesting pod/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 20:04:03.508000 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:03.507658 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 20:04:04.567204 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:04.567165 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerID="f644c3d16fda1d61f98310b27bae34516d3ebcd7365ec66cd3a60a1b9e33fe4c" exitCode=0 Jul 24 20:04:04.567740 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:04.567236 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" event={"ID":"3f5e1a44-fdf7-4a6c-8388-2c49d786a992","Type":"ContainerDied","Data":"f644c3d16fda1d61f98310b27bae34516d3ebcd7365ec66cd3a60a1b9e33fe4c"} Jul 24 20:04:04.567740 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:04.567723 2570 scope.go:117] "RemoveContainer" containerID="f644c3d16fda1d61f98310b27bae34516d3ebcd7365ec66cd3a60a1b9e33fe4c" Jul 24 20:04:04.568841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:04.568810 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:04:04.568936 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:04.568896 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:04:04.568999 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:04.568945 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:04:05.574707 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:05.574670 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" event={"ID":"3f5e1a44-fdf7-4a6c-8388-2c49d786a992","Type":"ContainerStarted","Data":"79589259078a0c1ed3b77aabf9a9c8d80825862ef9f7d9b7386e3f38c1aca1f0"} Jul 24 20:04:05.935955 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:05.935859 2570 patch_prober.go:28] interesting pod/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 24 20:04:05.936226 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:05.936198 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 24 20:04:05.936639 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:05.936618 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:04:06.935452 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:06.935413 2570 patch_prober.go:28] interesting pod/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 24 20:04:06.936092 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:06.936050 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 24 20:04:07.935272 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:07.935220 2570 patch_prober.go:28] interesting pod/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 24 20:04:07.935508 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:07.935304 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 24 20:04:08.935072 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:08.935032 2570 patch_prober.go:28] interesting pod/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 24 20:04:08.935342 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:08.935309 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 24 20:04:09.938408 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:09.938373 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:04:09.938813 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:09.938645 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:04:09.939400 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:09.939382 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:04:15.652929 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:15.652865 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:15.715449 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:15.715420 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:15.776334 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:15.776289 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:20.050662 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:20.050623 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:20.419584 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:20.419459 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:26.503984 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:26.503936 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:26.604630 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:26.604597 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:26.792103 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:26.792024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:27.002981 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:27.002950 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:42.570606 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.570566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:42.593611 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.593581 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x"] Jul 24 20:04:42.593813 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.593769 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" podUID="2a8b1631-1cca-47e4-a5cd-0cd855a16997" containerName="payload-processing" containerID="cri-o://3b2a77068d06aa182b3cfd79ab49065252ac47b0a8849d6d0e861bd17c090d0b" gracePeriod=30 Jul 24 20:04:42.670800 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.670760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:42.720148 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.720098 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:42.781595 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.781545 2570 generic.go:356] "Generic (PLEG): container finished" podID="2a8b1631-1cca-47e4-a5cd-0cd855a16997" containerID="3b2a77068d06aa182b3cfd79ab49065252ac47b0a8849d6d0e861bd17c090d0b" exitCode=0 Jul 24 20:04:42.781775 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.781672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" event={"ID":"2a8b1631-1cca-47e4-a5cd-0cd855a16997","Type":"ContainerDied","Data":"3b2a77068d06aa182b3cfd79ab49065252ac47b0a8849d6d0e861bd17c090d0b"} Jul 24 20:04:42.854450 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.854424 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:04:42.937540 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.937497 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8b1631-1cca-47e4-a5cd-0cd855a16997-plugins-config-volume\") pod \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " Jul 24 20:04:42.937726 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.937593 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkj7w\" (UniqueName: \"kubernetes.io/projected/2a8b1631-1cca-47e4-a5cd-0cd855a16997-kube-api-access-wkj7w\") pod \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\" (UID: \"2a8b1631-1cca-47e4-a5cd-0cd855a16997\") " Jul 24 20:04:42.937880 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.937857 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8b1631-1cca-47e4-a5cd-0cd855a16997-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a8b1631-1cca-47e4-a5cd-0cd855a16997" (UID: "2a8b1631-1cca-47e4-a5cd-0cd855a16997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 20:04:42.939620 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:42.939594 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8b1631-1cca-47e4-a5cd-0cd855a16997-kube-api-access-wkj7w" (OuterVolumeSpecName: "kube-api-access-wkj7w") pod "2a8b1631-1cca-47e4-a5cd-0cd855a16997" (UID: "2a8b1631-1cca-47e4-a5cd-0cd855a16997"). InnerVolumeSpecName "kube-api-access-wkj7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:04:43.038551 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:43.038517 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8b1631-1cca-47e4-a5cd-0cd855a16997-plugins-config-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:43.038551 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:43.038544 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkj7w\" (UniqueName: \"kubernetes.io/projected/2a8b1631-1cca-47e4-a5cd-0cd855a16997-kube-api-access-wkj7w\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:43.789244 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:43.789218 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" Jul 24 20:04:43.789716 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:43.789217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x" event={"ID":"2a8b1631-1cca-47e4-a5cd-0cd855a16997","Type":"ContainerDied","Data":"30ad650dcfb2eaf23cd32e2a7b62233a3e6b6a155532dac2b000d19a421b31c4"} Jul 24 20:04:43.789867 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:43.789778 2570 scope.go:117] "RemoveContainer" containerID="3b2a77068d06aa182b3cfd79ab49065252ac47b0a8849d6d0e861bd17c090d0b" Jul 24 20:04:43.812157 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:43.812136 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x"] Jul 24 20:04:43.814255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:43.814232 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e316a0-dd55c96c8-79h2x"] Jul 24 20:04:44.588149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:44.588116 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8b1631-1cca-47e4-a5cd-0cd855a16997" path="/var/lib/kubelet/pods/2a8b1631-1cca-47e4-a5cd-0cd855a16997/volumes" Jul 24 20:04:47.193291 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:47.193257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:47.589560 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:47.589498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:53.466556 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:53.466521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:53.567537 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:53.567506 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:53.714188 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:53.714155 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx"] Jul 24 20:04:53.714400 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:53.714370 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" containerID="cri-o://79589259078a0c1ed3b77aabf9a9c8d80825862ef9f7d9b7386e3f38c1aca1f0" gracePeriod=30 Jul 24 20:04:53.747997 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:53.747968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:53.957643 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:53.957610 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:54.325318 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:54.325291 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:54.633926 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:54.633842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:54.892727 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:54.892640 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:54.941845 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:54.941815 2570 patch_prober.go:28] interesting pod/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 20:04:54.941995 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:54.941877 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 20:04:57.787461 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:57.787390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:57.872348 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:57.872317 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:57.958635 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:57.958599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:58.308816 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:58.308785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:58.550334 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:58.550304 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:58.632899 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:58.632822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:04:58.874855 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:58.874822 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerID="79589259078a0c1ed3b77aabf9a9c8d80825862ef9f7d9b7386e3f38c1aca1f0" exitCode=0 Jul 24 20:04:58.875249 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:58.874899 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" event={"ID":"3f5e1a44-fdf7-4a6c-8388-2c49d786a992","Type":"ContainerDied","Data":"79589259078a0c1ed3b77aabf9a9c8d80825862ef9f7d9b7386e3f38c1aca1f0"} Jul 24 20:04:58.875249 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:58.874964 2570 scope.go:117] "RemoveContainer" containerID="f644c3d16fda1d61f98310b27bae34516d3ebcd7365ec66cd3a60a1b9e33fe4c" Jul 24 20:04:58.996761 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:58.996729 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:04:59.081686 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081657 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-credential-socket\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.081859 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081692 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-envoy\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.081859 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081717 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzl9s\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-kube-api-access-fzl9s\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.081859 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081766 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-data\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.081859 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081796 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istiod-ca-cert\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.081859 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081818 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-certs\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.082127 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081860 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-podinfo\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.082127 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081882 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-socket\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.082127 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081912 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-token\") pod \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\" (UID: \"3f5e1a44-fdf7-4a6c-8388-2c49d786a992\") " Jul 24 20:04:59.082127 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.081995 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:04:59.082331 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.082182 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 20:04:59.082331 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.082248 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:04:59.082331 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.082317 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istiod-ca-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.082941 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.082336 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-credential-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.082941 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.082462 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:04:59.082941 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.082710 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:04:59.084024 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.083994 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:04:59.084155 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.084017 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 20:04:59.084155 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.084104 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-kube-api-access-fzl9s" (OuterVolumeSpecName: "kube-api-access-fzl9s") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "kube-api-access-fzl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:04:59.084263 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.084247 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f5e1a44-fdf7-4a6c-8388-2c49d786a992" (UID: "3f5e1a44-fdf7-4a6c-8388-2c49d786a992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:04:59.182841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.182782 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-envoy\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.182841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.182802 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzl9s\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-kube-api-access-fzl9s\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.182841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.182812 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-data\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.182841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.182821 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-certs\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.182841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.182828 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-podinfo\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.182841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.182836 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-workload-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.182841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.182843 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5e1a44-fdf7-4a6c-8388-2c49d786a992-istio-token\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:04:59.881815 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.881777 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" event={"ID":"3f5e1a44-fdf7-4a6c-8388-2c49d786a992","Type":"ContainerDied","Data":"23f1f17dafded9f32ac4690a35ef08a0d1cbd5113a9ae23bdc518baff90cab28"} Jul 24 20:04:59.881815 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.881810 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx" Jul 24 20:04:59.882319 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.881849 2570 scope.go:117] "RemoveContainer" containerID="79589259078a0c1ed3b77aabf9a9c8d80825862ef9f7d9b7386e3f38c1aca1f0" Jul 24 20:04:59.904094 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.904067 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx"] Jul 24 20:04:59.910103 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:04:59.910080 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e316a0-openshift-default-c6c7576bb-9s9xx"] Jul 24 20:05:00.582702 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:00.582673 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" path="/var/lib/kubelet/pods/3f5e1a44-fdf7-4a6c-8388-2c49d786a992/volumes" Jul 24 20:05:02.268489 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.268447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:02.614941 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.614825 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:02.870220 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.870113 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m"] Jul 24 20:05:02.870825 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.870802 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8b1631-1cca-47e4-a5cd-0cd855a16997" containerName="payload-processing" Jul 24 20:05:02.870914 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.870829 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8b1631-1cca-47e4-a5cd-0cd855a16997" containerName="payload-processing" Jul 24 20:05:02.870914 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.870870 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" Jul 24 20:05:02.870914 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.870879 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" Jul 24 20:05:02.870914 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.870900 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" Jul 24 20:05:02.870914 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.870909 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" Jul 24 20:05:02.871172 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.871009 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" Jul 24 20:05:02.871172 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.871021 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8b1631-1cca-47e4-a5cd-0cd855a16997" containerName="payload-processing" Jul 24 20:05:02.875856 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.875836 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:02.878248 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.878225 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-3e15b4-openshift-default-dockercfg-x5z2r\"" Jul 24 20:05:02.886263 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.886238 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m"] Jul 24 20:05:02.902267 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:02.902232 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:03.020314 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-data\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020314 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-envoy\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020314 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020294 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-token\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020638 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-socket\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020638 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istiod-ca-cert\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020638 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-kube-api-access-h7djd\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020638 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-credential-socket\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020638 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020516 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-podinfo\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.020638 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.020558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-certs\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.121845 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.121754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istiod-ca-cert\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.121845 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.121797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-kube-api-access-h7djd\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.121845 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.121828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-credential-socket\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.121867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-podinfo\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.121893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-certs\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.121975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-data\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-envoy\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-token\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122149 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122079 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-socket\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122497 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122324 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-certs\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122497 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122371 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-credential-socket\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122676 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122647 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istiod-ca-cert\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122729 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122717 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-socket\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.122805 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.122782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-data\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.124423 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.124399 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-envoy\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.124572 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.124553 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-podinfo\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.131045 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.131018 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-kube-api-access-h7djd\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.131683 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.131658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-token\") pod \"e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.189440 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.189399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:03.335736 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.335706 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m"] Jul 24 20:05:03.338209 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:05:03.338182 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d08df4_ff68_4ed9_ae37_aef44bdc67a5.slice/crio-84b9aef1412e5ce674ae8caf63b08c7fea1feb3f5a714147368a16c0402b427e WatchSource:0}: Error finding container 84b9aef1412e5ce674ae8caf63b08c7fea1feb3f5a714147368a16c0402b427e: Status 404 returned error can't find the container with id 84b9aef1412e5ce674ae8caf63b08c7fea1feb3f5a714147368a16c0402b427e Jul 24 20:05:03.340858 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.340827 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:05:03.340956 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.340890 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:05:03.340956 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.340923 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:05:03.421342 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.421315 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:03.906210 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.906170 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" event={"ID":"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5","Type":"ContainerStarted","Data":"7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb"} Jul 24 20:05:03.906430 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.906228 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" event={"ID":"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5","Type":"ContainerStarted","Data":"84b9aef1412e5ce674ae8caf63b08c7fea1feb3f5a714147368a16c0402b427e"} Jul 24 20:05:03.927258 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:03.927201 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" podStartSLOduration=1.927185031 podStartE2EDuration="1.927185031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:05:03.925446214 +0000 UTC m=+2334.020935422" watchObservedRunningTime="2026-07-24 20:05:03.927185031 +0000 UTC m=+2334.022674238" Jul 24 20:05:04.190587 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:04.190508 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:04.195502 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:04.195459 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:04.912344 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:04.912312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:04.913331 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:04.913312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:05.842783 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:05.842750 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:05.943678 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:05.943641 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:06.854019 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:06.853985 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:07.081865 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:07.081834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:07.459723 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:07.459688 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:07.562943 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:07.562907 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:10.306918 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:10.306881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:24.015869 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:24.015835 2570 generic.go:356] "Generic (PLEG): container finished" podID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerID="7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb" exitCode=0 Jul 24 20:05:24.016298 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:24.015912 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" event={"ID":"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5","Type":"ContainerDied","Data":"7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb"} Jul 24 20:05:24.016342 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:24.016310 2570 scope.go:117] "RemoveContainer" containerID="7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb" Jul 24 20:05:24.017379 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:24.017346 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:05:24.017509 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:24.017436 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:05:24.017582 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:24.017514 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:05:25.023495 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:25.023436 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" event={"ID":"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5","Type":"ContainerStarted","Data":"cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0"} Jul 24 20:05:25.190557 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:25.190514 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:25.190832 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:25.190807 2570 patch_prober.go:28] interesting pod/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 24 20:05:25.190895 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:25.190864 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 24 20:05:26.190516 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:26.190457 2570 patch_prober.go:28] interesting pod/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 24 20:05:26.190916 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:26.190552 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 24 20:05:27.190429 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:27.190391 2570 patch_prober.go:28] interesting pod/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 24 20:05:27.190632 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:27.190457 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 24 20:05:28.189866 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:28.189827 2570 patch_prober.go:28] interesting pod/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 24 20:05:28.190077 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:28.189885 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 24 20:05:29.193696 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:29.193664 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:29.194109 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:29.194010 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:29.194707 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:29.194688 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:05:32.013539 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:32.013500 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:05:55.268869 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:05:55.268831 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:06:10.568755 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:06:10.568725 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:06:10.579843 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:06:10.579822 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:07:03.628704 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:07:03.628668 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:07:03.727737 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:07:03.727706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:07:20.575381 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:07:20.575340 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:07:58.993228 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:07:58.993193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:00.024814 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:00.024783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:11.620694 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:11.620661 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:25.083174 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:25.083135 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:25.181070 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:25.181035 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:29.520223 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:29.520182 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:45.405888 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:45.405833 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:45.505729 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:45.505699 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:56.831508 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:56.831450 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:56.933799 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:56.933760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:57.580013 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:57.579981 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:08:57.679395 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:08:57.679369 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:09:10.940228 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:09:10.940193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:09:11.722577 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:09:11.722543 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:09:12.046232 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:09:12.046200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:09:26.787968 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:09:26.787938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:09:51.327794 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:09:51.327759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:30.890821 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:30.890792 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:31.696973 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:31.696932 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:32.645864 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.645834 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7272-139819631123248"] Jul 24 20:10:32.646315 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.646301 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5e1a44-fdf7-4a6c-8388-2c49d786a992" containerName="istio-proxy" Jul 24 20:10:32.649362 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.649344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123248" Jul 24 20:10:32.652148 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.652101 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q9hpc\"" Jul 24 20:10:32.655831 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.655804 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631123248"] Jul 24 20:10:32.775961 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.775916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58tr\" (UniqueName: \"kubernetes.io/projected/9fc0ef0b-bb62-4953-be40-de952b8768a4-kube-api-access-f58tr\") pod \"test-curl-7272-139819631123248\" (UID: \"9fc0ef0b-bb62-4953-be40-de952b8768a4\") " pod="opendatahub/test-curl-7272-139819631123248" Jul 24 20:10:32.876777 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.876746 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58tr\" (UniqueName: \"kubernetes.io/projected/9fc0ef0b-bb62-4953-be40-de952b8768a4-kube-api-access-f58tr\") pod \"test-curl-7272-139819631123248\" (UID: \"9fc0ef0b-bb62-4953-be40-de952b8768a4\") " pod="opendatahub/test-curl-7272-139819631123248" Jul 24 20:10:32.885568 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.885544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58tr\" (UniqueName: \"kubernetes.io/projected/9fc0ef0b-bb62-4953-be40-de952b8768a4-kube-api-access-f58tr\") pod \"test-curl-7272-139819631123248\" (UID: \"9fc0ef0b-bb62-4953-be40-de952b8768a4\") " pod="opendatahub/test-curl-7272-139819631123248" Jul 24 20:10:32.960330 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:32.960264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123248" Jul 24 20:10:33.095501 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:33.095461 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631123248"] Jul 24 20:10:33.098014 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:10:33.097981 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fc0ef0b_bb62_4953_be40_de952b8768a4.slice/crio-46adec8f784df3968e4e639c8a7970a61ec42fbae6ba03db27e5b1c0ddb2200e WatchSource:0}: Error finding container 46adec8f784df3968e4e639c8a7970a61ec42fbae6ba03db27e5b1c0ddb2200e: Status 404 returned error can't find the container with id 46adec8f784df3968e4e639c8a7970a61ec42fbae6ba03db27e5b1c0ddb2200e Jul 24 20:10:33.099342 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:33.099322 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 20:10:33.650524 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:33.650456 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631123248" event={"ID":"9fc0ef0b-bb62-4953-be40-de952b8768a4","Type":"ContainerStarted","Data":"46adec8f784df3968e4e639c8a7970a61ec42fbae6ba03db27e5b1c0ddb2200e"} Jul 24 20:10:34.657661 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:34.657626 2570 generic.go:356] "Generic (PLEG): container finished" podID="9fc0ef0b-bb62-4953-be40-de952b8768a4" containerID="5a214063c0dba10e9bf8a93c6d74f51379f4e0921866a80469e02596596da2b1" exitCode=0 Jul 24 20:10:34.658084 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:34.657717 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631123248" event={"ID":"9fc0ef0b-bb62-4953-be40-de952b8768a4","Type":"ContainerDied","Data":"5a214063c0dba10e9bf8a93c6d74f51379f4e0921866a80469e02596596da2b1"} Jul 24 20:10:35.803665 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.803643 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123248" Jul 24 20:10:35.825818 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.825793 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7272-139819631123248_9fc0ef0b-bb62-4953-be40-de952b8768a4/test-curl-7272-139819631123248/0.log" Jul 24 20:10:35.850198 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.850173 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7272-139819631123248"] Jul 24 20:10:35.862084 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.862056 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7272-139819631123248"] Jul 24 20:10:35.906542 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.906513 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f58tr\" (UniqueName: \"kubernetes.io/projected/9fc0ef0b-bb62-4953-be40-de952b8768a4-kube-api-access-f58tr\") pod \"9fc0ef0b-bb62-4953-be40-de952b8768a4\" (UID: \"9fc0ef0b-bb62-4953-be40-de952b8768a4\") " Jul 24 20:10:35.908505 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.908457 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc0ef0b-bb62-4953-be40-de952b8768a4-kube-api-access-f58tr" (OuterVolumeSpecName: "kube-api-access-f58tr") pod "9fc0ef0b-bb62-4953-be40-de952b8768a4" (UID: "9fc0ef0b-bb62-4953-be40-de952b8768a4"). InnerVolumeSpecName "kube-api-access-f58tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:10:35.973388 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.973360 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7272-139819631123120"] Jul 24 20:10:35.973794 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.973780 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc0ef0b-bb62-4953-be40-de952b8768a4" containerName="test-curl-7272-139819631123248" Jul 24 20:10:35.973843 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.973795 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc0ef0b-bb62-4953-be40-de952b8768a4" containerName="test-curl-7272-139819631123248" Jul 24 20:10:35.973889 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.973880 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc0ef0b-bb62-4953-be40-de952b8768a4" containerName="test-curl-7272-139819631123248" Jul 24 20:10:35.976948 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.976934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:35.981317 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:35.981294 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631123120"] Jul 24 20:10:36.007184 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.007160 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58tr\" (UniqueName: \"kubernetes.io/projected/9fc0ef0b-bb62-4953-be40-de952b8768a4-kube-api-access-f58tr\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:10:36.107799 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.107773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qcct\" (UniqueName: \"kubernetes.io/projected/b39c930e-2d34-4f3d-9784-ef41ead16b67-kube-api-access-5qcct\") pod \"test-curl-7272-139819631123120\" (UID: \"b39c930e-2d34-4f3d-9784-ef41ead16b67\") " pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:36.208679 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.208604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qcct\" (UniqueName: \"kubernetes.io/projected/b39c930e-2d34-4f3d-9784-ef41ead16b67-kube-api-access-5qcct\") pod \"test-curl-7272-139819631123120\" (UID: \"b39c930e-2d34-4f3d-9784-ef41ead16b67\") " pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:36.221564 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.221538 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qcct\" (UniqueName: \"kubernetes.io/projected/b39c930e-2d34-4f3d-9784-ef41ead16b67-kube-api-access-5qcct\") pod \"test-curl-7272-139819631123120\" (UID: \"b39c930e-2d34-4f3d-9784-ef41ead16b67\") " pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:36.288036 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.288012 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:36.581737 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.581703 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc0ef0b-bb62-4953-be40-de952b8768a4" path="/var/lib/kubelet/pods/9fc0ef0b-bb62-4953-be40-de952b8768a4/volumes" Jul 24 20:10:36.629610 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.629585 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631123120"] Jul 24 20:10:36.631366 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:10:36.631334 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb39c930e_2d34_4f3d_9784_ef41ead16b67.slice/crio-06e2fd9b0d53b14d9d546e64b2ab2c93546541eca0cf06dda20cf51e99b56089 WatchSource:0}: Error finding container 06e2fd9b0d53b14d9d546e64b2ab2c93546541eca0cf06dda20cf51e99b56089: Status 404 returned error can't find the container with id 06e2fd9b0d53b14d9d546e64b2ab2c93546541eca0cf06dda20cf51e99b56089 Jul 24 20:10:36.669763 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.669733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631123120" event={"ID":"b39c930e-2d34-4f3d-9784-ef41ead16b67","Type":"ContainerStarted","Data":"06e2fd9b0d53b14d9d546e64b2ab2c93546541eca0cf06dda20cf51e99b56089"} Jul 24 20:10:36.670925 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.670909 2570 scope.go:117] "RemoveContainer" containerID="5a214063c0dba10e9bf8a93c6d74f51379f4e0921866a80469e02596596da2b1" Jul 24 20:10:36.671035 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:36.670911 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123248" Jul 24 20:10:37.678601 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:37.678567 2570 generic.go:356] "Generic (PLEG): container finished" podID="b39c930e-2d34-4f3d-9784-ef41ead16b67" containerID="ba7222ca16cd882a149fed27f6f4d30c5fcf17829a868a3bbb11d88c966128ca" exitCode=0 Jul 24 20:10:37.678990 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:37.678617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631123120" event={"ID":"b39c930e-2d34-4f3d-9784-ef41ead16b67","Type":"ContainerDied","Data":"ba7222ca16cd882a149fed27f6f4d30c5fcf17829a868a3bbb11d88c966128ca"} Jul 24 20:10:38.827587 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:38.827564 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:38.846141 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:38.846117 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7272-139819631123120_b39c930e-2d34-4f3d-9784-ef41ead16b67/test-curl-7272-139819631123120/0.log" Jul 24 20:10:38.866162 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:38.866134 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7272-139819631123120"] Jul 24 20:10:38.868225 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:38.868200 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7272-139819631123120"] Jul 24 20:10:38.934718 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:38.934650 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qcct\" (UniqueName: \"kubernetes.io/projected/b39c930e-2d34-4f3d-9784-ef41ead16b67-kube-api-access-5qcct\") pod \"b39c930e-2d34-4f3d-9784-ef41ead16b67\" (UID: \"b39c930e-2d34-4f3d-9784-ef41ead16b67\") " Jul 24 20:10:38.936645 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:38.936619 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39c930e-2d34-4f3d-9784-ef41ead16b67-kube-api-access-5qcct" (OuterVolumeSpecName: "kube-api-access-5qcct") pod "b39c930e-2d34-4f3d-9784-ef41ead16b67" (UID: "b39c930e-2d34-4f3d-9784-ef41ead16b67"). InnerVolumeSpecName "kube-api-access-5qcct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:10:39.027778 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.027748 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:39.028199 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.028187 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39c930e-2d34-4f3d-9784-ef41ead16b67" containerName="test-curl-7272-139819631123120" Jul 24 20:10:39.028249 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.028201 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39c930e-2d34-4f3d-9784-ef41ead16b67" containerName="test-curl-7272-139819631123120" Jul 24 20:10:39.028285 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.028271 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b39c930e-2d34-4f3d-9784-ef41ead16b67" containerName="test-curl-7272-139819631123120" Jul 24 20:10:39.031245 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.031230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:39.035698 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.035670 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qcct\" (UniqueName: \"kubernetes.io/projected/b39c930e-2d34-4f3d-9784-ef41ead16b67-kube-api-access-5qcct\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:10:39.037598 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.037579 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:39.137177 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.137146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56ks\" (UniqueName: \"kubernetes.io/projected/3ab0744c-688f-42ce-81e1-3f4f6e2d5365-kube-api-access-t56ks\") pod \"test-curl-7272-139819637540880\" (UID: \"3ab0744c-688f-42ce-81e1-3f4f6e2d5365\") " pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:39.238099 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.238063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56ks\" (UniqueName: \"kubernetes.io/projected/3ab0744c-688f-42ce-81e1-3f4f6e2d5365-kube-api-access-t56ks\") pod \"test-curl-7272-139819637540880\" (UID: \"3ab0744c-688f-42ce-81e1-3f4f6e2d5365\") " pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:39.245666 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.245643 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56ks\" (UniqueName: \"kubernetes.io/projected/3ab0744c-688f-42ce-81e1-3f4f6e2d5365-kube-api-access-t56ks\") pod \"test-curl-7272-139819637540880\" (UID: \"3ab0744c-688f-42ce-81e1-3f4f6e2d5365\") " pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:39.341167 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.341144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:39.483656 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.483633 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:39.485203 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:10:39.485176 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ab0744c_688f_42ce_81e1_3f4f6e2d5365.slice/crio-9c9d644132b902f4c5a74827766aa5103ff1dfafccbbdf1c721d7fc9a06854bf WatchSource:0}: Error finding container 9c9d644132b902f4c5a74827766aa5103ff1dfafccbbdf1c721d7fc9a06854bf: Status 404 returned error can't find the container with id 9c9d644132b902f4c5a74827766aa5103ff1dfafccbbdf1c721d7fc9a06854bf Jul 24 20:10:39.689054 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.689027 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:39.689228 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.689029 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e2fd9b0d53b14d9d546e64b2ab2c93546541eca0cf06dda20cf51e99b56089" Jul 24 20:10:39.690342 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:39.690319 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819637540880" event={"ID":"3ab0744c-688f-42ce-81e1-3f4f6e2d5365","Type":"ContainerStarted","Data":"9c9d644132b902f4c5a74827766aa5103ff1dfafccbbdf1c721d7fc9a06854bf"} Jul 24 20:10:39.703211 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:10:39.703183 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7272-139819631123120\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="b39c930e-2d34-4f3d-9784-ef41ead16b67" pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:40.579993 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:10:40.579958 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7272-139819631123120\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="b39c930e-2d34-4f3d-9784-ef41ead16b67" pod="opendatahub/test-curl-7272-139819631123120" Jul 24 20:10:40.582339 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:40.582315 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b39c930e-2d34-4f3d-9784-ef41ead16b67" path="/var/lib/kubelet/pods/b39c930e-2d34-4f3d-9784-ef41ead16b67/volumes" Jul 24 20:10:40.696601 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:40.696571 2570 generic.go:356] "Generic (PLEG): container finished" podID="3ab0744c-688f-42ce-81e1-3f4f6e2d5365" containerID="ef3e02451d7a2adfc00e49f6b581da6e4921d8fc17eb74b3088fed04479fe685" exitCode=0 Jul 24 20:10:40.696750 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:40.696654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819637540880" event={"ID":"3ab0744c-688f-42ce-81e1-3f4f6e2d5365","Type":"ContainerDied","Data":"ef3e02451d7a2adfc00e49f6b581da6e4921d8fc17eb74b3088fed04479fe685"} Jul 24 20:10:41.850021 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:41.849995 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:41.870713 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:41.870691 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7272-139819637540880_3ab0744c-688f-42ce-81e1-3f4f6e2d5365/test-curl-7272-139819637540880/0.log" Jul 24 20:10:41.893696 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:41.893669 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:41.897591 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:41.897569 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:41.963962 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:41.963907 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t56ks\" (UniqueName: \"kubernetes.io/projected/3ab0744c-688f-42ce-81e1-3f4f6e2d5365-kube-api-access-t56ks\") pod \"3ab0744c-688f-42ce-81e1-3f4f6e2d5365\" (UID: \"3ab0744c-688f-42ce-81e1-3f4f6e2d5365\") " Jul 24 20:10:41.965971 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:41.965946 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab0744c-688f-42ce-81e1-3f4f6e2d5365-kube-api-access-t56ks" (OuterVolumeSpecName: "kube-api-access-t56ks") pod "3ab0744c-688f-42ce-81e1-3f4f6e2d5365" (UID: "3ab0744c-688f-42ce-81e1-3f4f6e2d5365"). InnerVolumeSpecName "kube-api-access-t56ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:10:42.019437 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.019411 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:42.019855 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.019842 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab0744c-688f-42ce-81e1-3f4f6e2d5365" containerName="test-curl-7272-139819637540880" Jul 24 20:10:42.019909 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.019857 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab0744c-688f-42ce-81e1-3f4f6e2d5365" containerName="test-curl-7272-139819637540880" Jul 24 20:10:42.019946 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.019930 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ab0744c-688f-42ce-81e1-3f4f6e2d5365" containerName="test-curl-7272-139819637540880" Jul 24 20:10:42.022885 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.022870 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:42.030652 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.030621 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:42.064902 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.064876 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56ks\" (UniqueName: \"kubernetes.io/projected/3ab0744c-688f-42ce-81e1-3f4f6e2d5365-kube-api-access-t56ks\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:10:42.165476 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.165445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpsb8\" (UniqueName: \"kubernetes.io/projected/d363da4d-aeba-4e63-9c72-a1510e5d0a4d-kube-api-access-dpsb8\") pod \"test-curl-7272-139819637540880\" (UID: \"d363da4d-aeba-4e63-9c72-a1510e5d0a4d\") " pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:42.266870 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.266842 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsb8\" (UniqueName: \"kubernetes.io/projected/d363da4d-aeba-4e63-9c72-a1510e5d0a4d-kube-api-access-dpsb8\") pod \"test-curl-7272-139819637540880\" (UID: \"d363da4d-aeba-4e63-9c72-a1510e5d0a4d\") " pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:42.274227 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.274204 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpsb8\" (UniqueName: \"kubernetes.io/projected/d363da4d-aeba-4e63-9c72-a1510e5d0a4d-kube-api-access-dpsb8\") pod \"test-curl-7272-139819637540880\" (UID: \"d363da4d-aeba-4e63-9c72-a1510e5d0a4d\") " pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:42.333347 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.333312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:42.466505 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.466458 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:42.468252 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:10:42.468227 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd363da4d_aeba_4e63_9c72_a1510e5d0a4d.slice/crio-8af27cf2de7f6ac7fa4d4d82ddeae6cfe8f425fe24d467f2787c1c9ecf6fcaf1 WatchSource:0}: Error finding container 8af27cf2de7f6ac7fa4d4d82ddeae6cfe8f425fe24d467f2787c1c9ecf6fcaf1: Status 404 returned error can't find the container with id 8af27cf2de7f6ac7fa4d4d82ddeae6cfe8f425fe24d467f2787c1c9ecf6fcaf1 Jul 24 20:10:42.581542 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.581441 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ab0744c-688f-42ce-81e1-3f4f6e2d5365" path="/var/lib/kubelet/pods/3ab0744c-688f-42ce-81e1-3f4f6e2d5365/volumes" Jul 24 20:10:42.708285 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.708259 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:42.708285 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.708284 2570 scope.go:117] "RemoveContainer" containerID="ef3e02451d7a2adfc00e49f6b581da6e4921d8fc17eb74b3088fed04479fe685" Jul 24 20:10:42.709747 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:42.709698 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819637540880" event={"ID":"d363da4d-aeba-4e63-9c72-a1510e5d0a4d","Type":"ContainerStarted","Data":"8af27cf2de7f6ac7fa4d4d82ddeae6cfe8f425fe24d467f2787c1c9ecf6fcaf1"} Jul 24 20:10:43.716591 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:43.716559 2570 generic.go:356] "Generic (PLEG): container finished" podID="d363da4d-aeba-4e63-9c72-a1510e5d0a4d" containerID="5e871ebb86b78f9505dfcc41567cdf26da579142698fcd776843070c2f614b51" exitCode=0 Jul 24 20:10:43.717001 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:43.716645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819637540880" event={"ID":"d363da4d-aeba-4e63-9c72-a1510e5d0a4d","Type":"ContainerDied","Data":"5e871ebb86b78f9505dfcc41567cdf26da579142698fcd776843070c2f614b51"} Jul 24 20:10:44.868813 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:44.868788 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:44.887372 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:44.887339 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7272-139819637540880_d363da4d-aeba-4e63-9c72-a1510e5d0a4d/test-curl-7272-139819637540880/0.log" Jul 24 20:10:44.910581 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:44.910557 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:44.914036 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:44.914016 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7272-139819637540880"] Jul 24 20:10:44.992408 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:44.992385 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpsb8\" (UniqueName: \"kubernetes.io/projected/d363da4d-aeba-4e63-9c72-a1510e5d0a4d-kube-api-access-dpsb8\") pod \"d363da4d-aeba-4e63-9c72-a1510e5d0a4d\" (UID: \"d363da4d-aeba-4e63-9c72-a1510e5d0a4d\") " Jul 24 20:10:44.994420 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:44.994395 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d363da4d-aeba-4e63-9c72-a1510e5d0a4d-kube-api-access-dpsb8" (OuterVolumeSpecName: "kube-api-access-dpsb8") pod "d363da4d-aeba-4e63-9c72-a1510e5d0a4d" (UID: "d363da4d-aeba-4e63-9c72-a1510e5d0a4d"). InnerVolumeSpecName "kube-api-access-dpsb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:10:45.037636 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.037613 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:45.038086 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.038072 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d363da4d-aeba-4e63-9c72-a1510e5d0a4d" containerName="test-curl-7272-139819637540880" Jul 24 20:10:45.038131 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.038087 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d363da4d-aeba-4e63-9c72-a1510e5d0a4d" containerName="test-curl-7272-139819637540880" Jul 24 20:10:45.038178 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.038168 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d363da4d-aeba-4e63-9c72-a1510e5d0a4d" containerName="test-curl-7272-139819637540880" Jul 24 20:10:45.041210 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.041195 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:45.048492 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.048457 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:45.093606 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.093578 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpsb8\" (UniqueName: \"kubernetes.io/projected/d363da4d-aeba-4e63-9c72-a1510e5d0a4d-kube-api-access-dpsb8\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:10:45.194519 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.194492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ntz5\" (UniqueName: \"kubernetes.io/projected/580e9325-56c3-4164-9a71-bd885504bbcc-kube-api-access-6ntz5\") pod \"test-curl-7272-139819631076816\" (UID: \"580e9325-56c3-4164-9a71-bd885504bbcc\") " pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:45.295947 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.295878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ntz5\" (UniqueName: \"kubernetes.io/projected/580e9325-56c3-4164-9a71-bd885504bbcc-kube-api-access-6ntz5\") pod \"test-curl-7272-139819631076816\" (UID: \"580e9325-56c3-4164-9a71-bd885504bbcc\") " pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:45.303000 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.302979 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ntz5\" (UniqueName: \"kubernetes.io/projected/580e9325-56c3-4164-9a71-bd885504bbcc-kube-api-access-6ntz5\") pod \"test-curl-7272-139819631076816\" (UID: \"580e9325-56c3-4164-9a71-bd885504bbcc\") " pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:45.351836 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.351804 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:45.482346 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.482319 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:45.484305 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:10:45.484277 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod580e9325_56c3_4164_9a71_bd885504bbcc.slice/crio-fc7aa6910bd6ca97ee2919a069ad26f350633aaf43788b65442d842779fb8013 WatchSource:0}: Error finding container fc7aa6910bd6ca97ee2919a069ad26f350633aaf43788b65442d842779fb8013: Status 404 returned error can't find the container with id fc7aa6910bd6ca97ee2919a069ad26f350633aaf43788b65442d842779fb8013 Jul 24 20:10:45.733048 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.732953 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:45.733048 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.732998 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af27cf2de7f6ac7fa4d4d82ddeae6cfe8f425fe24d467f2787c1c9ecf6fcaf1" Jul 24 20:10:45.734287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:45.734256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631076816" event={"ID":"580e9325-56c3-4164-9a71-bd885504bbcc","Type":"ContainerStarted","Data":"fc7aa6910bd6ca97ee2919a069ad26f350633aaf43788b65442d842779fb8013"} Jul 24 20:10:45.743969 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:10:45.743944 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7272-139819637540880\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="d363da4d-aeba-4e63-9c72-a1510e5d0a4d" pod="opendatahub/test-curl-7272-139819637540880" Jul 24 20:10:46.583055 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:46.583025 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d363da4d-aeba-4e63-9c72-a1510e5d0a4d" path="/var/lib/kubelet/pods/d363da4d-aeba-4e63-9c72-a1510e5d0a4d/volumes" Jul 24 20:10:46.740818 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:46.740785 2570 generic.go:356] "Generic (PLEG): container finished" podID="580e9325-56c3-4164-9a71-bd885504bbcc" containerID="7f40a5ac11c551da2a368351988af3a421f2af713fe78847f85571d0165eced5" exitCode=0 Jul 24 20:10:46.740970 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:46.740855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631076816" event={"ID":"580e9325-56c3-4164-9a71-bd885504bbcc","Type":"ContainerDied","Data":"7f40a5ac11c551da2a368351988af3a421f2af713fe78847f85571d0165eced5"} Jul 24 20:10:47.891702 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:47.891676 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:47.911869 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:47.911844 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7272-139819631076816_580e9325-56c3-4164-9a71-bd885504bbcc/test-curl-7272-139819631076816/0.log" Jul 24 20:10:47.920289 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:47.920269 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ntz5\" (UniqueName: \"kubernetes.io/projected/580e9325-56c3-4164-9a71-bd885504bbcc-kube-api-access-6ntz5\") pod \"580e9325-56c3-4164-9a71-bd885504bbcc\" (UID: \"580e9325-56c3-4164-9a71-bd885504bbcc\") " Jul 24 20:10:47.922296 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:47.922265 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580e9325-56c3-4164-9a71-bd885504bbcc-kube-api-access-6ntz5" (OuterVolumeSpecName: "kube-api-access-6ntz5") pod "580e9325-56c3-4164-9a71-bd885504bbcc" (UID: "580e9325-56c3-4164-9a71-bd885504bbcc"). InnerVolumeSpecName "kube-api-access-6ntz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:10:47.934775 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:47.934750 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:47.936738 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:47.936719 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:48.021892 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.021834 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ntz5\" (UniqueName: \"kubernetes.io/projected/580e9325-56c3-4164-9a71-bd885504bbcc-kube-api-access-6ntz5\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:10:48.058972 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.058950 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:48.059343 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.059331 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580e9325-56c3-4164-9a71-bd885504bbcc" containerName="test-curl-7272-139819631076816" Jul 24 20:10:48.059391 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.059344 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="580e9325-56c3-4164-9a71-bd885504bbcc" containerName="test-curl-7272-139819631076816" Jul 24 20:10:48.059435 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.059426 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="580e9325-56c3-4164-9a71-bd885504bbcc" containerName="test-curl-7272-139819631076816" Jul 24 20:10:48.062394 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.062380 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:48.068694 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.068671 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:48.122935 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.122907 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4mt\" (UniqueName: \"kubernetes.io/projected/58e8e011-ae71-4cbd-bcd1-7a33f9ee4655-kube-api-access-tk4mt\") pod \"test-curl-7272-139819631076816\" (UID: \"58e8e011-ae71-4cbd-bcd1-7a33f9ee4655\") " pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:48.223348 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.223323 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4mt\" (UniqueName: \"kubernetes.io/projected/58e8e011-ae71-4cbd-bcd1-7a33f9ee4655-kube-api-access-tk4mt\") pod \"test-curl-7272-139819631076816\" (UID: \"58e8e011-ae71-4cbd-bcd1-7a33f9ee4655\") " pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:48.231040 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.231020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4mt\" (UniqueName: \"kubernetes.io/projected/58e8e011-ae71-4cbd-bcd1-7a33f9ee4655-kube-api-access-tk4mt\") pod \"test-curl-7272-139819631076816\" (UID: \"58e8e011-ae71-4cbd-bcd1-7a33f9ee4655\") " pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:48.374255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.374186 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:48.519897 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.516011 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:48.519897 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:10:48.516597 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e8e011_ae71_4cbd_bcd1_7a33f9ee4655.slice/crio-4864bfb647e10d22b62f702c8fdc9df2f456d73e7d716424f7c33c934d9bf832 WatchSource:0}: Error finding container 4864bfb647e10d22b62f702c8fdc9df2f456d73e7d716424f7c33c934d9bf832: Status 404 returned error can't find the container with id 4864bfb647e10d22b62f702c8fdc9df2f456d73e7d716424f7c33c934d9bf832 Jul 24 20:10:48.582223 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.582198 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580e9325-56c3-4164-9a71-bd885504bbcc" path="/var/lib/kubelet/pods/580e9325-56c3-4164-9a71-bd885504bbcc/volumes" Jul 24 20:10:48.752790 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.752759 2570 scope.go:117] "RemoveContainer" containerID="7f40a5ac11c551da2a368351988af3a421f2af713fe78847f85571d0165eced5" Jul 24 20:10:48.752958 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.752763 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:48.754161 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:48.754131 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631076816" event={"ID":"58e8e011-ae71-4cbd-bcd1-7a33f9ee4655","Type":"ContainerStarted","Data":"4864bfb647e10d22b62f702c8fdc9df2f456d73e7d716424f7c33c934d9bf832"} Jul 24 20:10:49.761863 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:49.761832 2570 generic.go:356] "Generic (PLEG): container finished" podID="58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" containerID="da118d244e3e92211015a209bc5b88fde9d4f5f41a442d2701399ff8639b3223" exitCode=0 Jul 24 20:10:49.762265 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:49.761892 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7272-139819631076816" event={"ID":"58e8e011-ae71-4cbd-bcd1-7a33f9ee4655","Type":"ContainerDied","Data":"da118d244e3e92211015a209bc5b88fde9d4f5f41a442d2701399ff8639b3223"} Jul 24 20:10:50.916539 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:50.916511 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:50.938813 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:50.938785 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7272-139819631076816_58e8e011-ae71-4cbd-bcd1-7a33f9ee4655/test-curl-7272-139819631076816/0.log" Jul 24 20:10:50.948005 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:50.947975 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk4mt\" (UniqueName: \"kubernetes.io/projected/58e8e011-ae71-4cbd-bcd1-7a33f9ee4655-kube-api-access-tk4mt\") pod \"58e8e011-ae71-4cbd-bcd1-7a33f9ee4655\" (UID: \"58e8e011-ae71-4cbd-bcd1-7a33f9ee4655\") " Jul 24 20:10:50.950424 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:50.950385 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e8e011-ae71-4cbd-bcd1-7a33f9ee4655-kube-api-access-tk4mt" (OuterVolumeSpecName: "kube-api-access-tk4mt") pod "58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" (UID: "58e8e011-ae71-4cbd-bcd1-7a33f9ee4655"). InnerVolumeSpecName "kube-api-access-tk4mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:10:50.962371 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:50.962339 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:50.965626 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:50.965585 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7272-139819631076816"] Jul 24 20:10:51.049933 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.049842 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk4mt\" (UniqueName: \"kubernetes.io/projected/58e8e011-ae71-4cbd-bcd1-7a33f9ee4655-kube-api-access-tk4mt\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:10:51.480481 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.480383 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:51.774980 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.774955 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4864bfb647e10d22b62f702c8fdc9df2f456d73e7d716424f7c33c934d9bf832" Jul 24 20:10:51.774980 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.774964 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:51.785754 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:10:51.785725 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7272-139819631076816\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:51.838387 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.838352 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79"] Jul 24 20:10:51.838851 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.838832 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" containerName="test-curl-7272-139819631076816" Jul 24 20:10:51.838939 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.838853 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" containerName="test-curl-7272-139819631076816" Jul 24 20:10:51.839036 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.839022 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" containerName="test-curl-7272-139819631076816" Jul 24 20:10:51.842057 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.842034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.844382 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:10:51.844345 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7272-139819631076816\" is forbidden: User \"system:node:ip-10-0-141-238.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-238.ec2.internal' and this object" podUID="58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" pod="opendatahub/test-curl-7272-139819631076816" Jul 24 20:10:51.844532 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.844433 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-11ccc9-openshift-default-dockercfg-bngfb\"" Jul 24 20:10:51.848997 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.848946 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.859417 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79"] Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860351 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-data\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsbh\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-kube-api-access-zrsbh\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860454 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-credential-socket\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860502 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-envoy\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-token\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860612 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-podinfo\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860643 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-certs\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860731 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-socket\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.864731 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.860771 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b90c562-9ae4-4d34-a577-32ae904db2e3-istiod-ca-cert\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.961899 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.961866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b90c562-9ae4-4d34-a577-32ae904db2e3-istiod-ca-cert\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.961909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-data\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.961938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsbh\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-kube-api-access-zrsbh\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.961957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-credential-socket\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.961971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-envoy\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-token\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962051 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-podinfo\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962077 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-certs\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962364 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-socket\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962846 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962446 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-data\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962846 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962532 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-credential-socket\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962846 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962623 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-certs\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962846 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-socket\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.962846 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.962797 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b90c562-9ae4-4d34-a577-32ae904db2e3-istiod-ca-cert\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.964672 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.964643 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-envoy\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.964902 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.964883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-podinfo\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.973138 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.973112 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-token\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:51.973538 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:51.973515 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsbh\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-kube-api-access-zrsbh\") pod \"e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:52.025652 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.025574 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:52.153999 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.153962 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:52.302163 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.302135 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79"] Jul 24 20:10:52.303983 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:10:52.303952 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b90c562_9ae4_4d34_a577_32ae904db2e3.slice/crio-f29d666705d391ed7401adf305844f04ec0664e9453c81e13837fb5e5ba50503 WatchSource:0}: Error finding container f29d666705d391ed7401adf305844f04ec0664e9453c81e13837fb5e5ba50503: Status 404 returned error can't find the container with id f29d666705d391ed7401adf305844f04ec0664e9453c81e13837fb5e5ba50503 Jul 24 20:10:52.306439 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.306386 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:10:52.306529 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.306483 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:10:52.306529 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.306512 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:10:52.583064 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.582983 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e8e011-ae71-4cbd-bcd1-7a33f9ee4655" path="/var/lib/kubelet/pods/58e8e011-ae71-4cbd-bcd1-7a33f9ee4655/volumes" Jul 24 20:10:52.781774 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.781742 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" event={"ID":"0b90c562-9ae4-4d34-a577-32ae904db2e3","Type":"ContainerStarted","Data":"683d88897716d0ef89e6086515517a764988e24107c529d16d857cf3efe0ad38"} Jul 24 20:10:52.781927 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.781799 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" event={"ID":"0b90c562-9ae4-4d34-a577-32ae904db2e3","Type":"ContainerStarted","Data":"f29d666705d391ed7401adf305844f04ec0664e9453c81e13837fb5e5ba50503"} Jul 24 20:10:52.802033 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:52.801977 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podStartSLOduration=1.801956413 podStartE2EDuration="1.801956413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:10:52.799520573 +0000 UTC m=+2682.895009778" watchObservedRunningTime="2026-07-24 20:10:52.801956413 +0000 UTC m=+2682.897445621" Jul 24 20:10:53.154441 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:53.154406 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:53.156108 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:53.156081 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 24 20:10:53.156238 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:53.156131 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 24 20:10:54.159125 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:54.159093 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:54.741519 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:54.741463 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:54.792988 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:54.792951 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:54.793851 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:54.793834 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:10:54.820702 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:54.820675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:55.931853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:55.931818 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:56.031157 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:56.031125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:56.402440 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:56.402407 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:56.505862 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:56.505828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:10:59.275912 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:10:59.275879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:10.198914 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:10.198869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:10.522372 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:10.522333 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:10.625655 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:10.625630 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:11:10.628045 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:10.628019 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:11:10.738927 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:10.738875 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:10.833699 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:10.833609 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 20:11:10.833699 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:10.833675 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 20:11:12.897055 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:12.897013 2570 generic.go:356] "Generic (PLEG): container finished" podID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerID="683d88897716d0ef89e6086515517a764988e24107c529d16d857cf3efe0ad38" exitCode=0 Jul 24 20:11:12.897484 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:12.897083 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" event={"ID":"0b90c562-9ae4-4d34-a577-32ae904db2e3","Type":"ContainerDied","Data":"683d88897716d0ef89e6086515517a764988e24107c529d16d857cf3efe0ad38"} Jul 24 20:11:12.897548 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:12.897533 2570 scope.go:117] "RemoveContainer" containerID="683d88897716d0ef89e6086515517a764988e24107c529d16d857cf3efe0ad38" Jul 24 20:11:12.898719 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:12.898692 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:11:12.898800 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:12.898763 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:11:12.898800 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:12.898792 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 20:11:13.904515 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:13.904479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" event={"ID":"0b90c562-9ae4-4d34-a577-32ae904db2e3","Type":"ContainerStarted","Data":"55dbfb246400ae2ddff396a5ea485fc0874b3e1207a495bb352de8bc2d91a940"} Jul 24 20:11:14.155585 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:14.155451 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:11:14.155888 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:14.155540 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 24 20:11:14.156042 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:14.156020 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 24 20:11:15.154877 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:15.154839 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 24 20:11:15.155486 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:15.155433 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 24 20:11:16.154885 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:16.154846 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 24 20:11:16.155519 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:16.155457 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 24 20:11:16.641075 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:16.641021 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:16.725053 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:16.725001 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:17.155074 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:17.154638 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 24 20:11:17.155074 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:17.154702 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 24 20:11:18.158263 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:18.158231 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:11:18.158667 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:18.158574 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:11:18.159138 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:18.159119 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:11:18.585070 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:18.585043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:18.873956 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:18.873874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:19.137206 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:19.137131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:19.242849 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:19.242821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:21.142645 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:21.142589 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:11:48.579902 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:11:48.579866 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cxnzs_ecc30144-c524-45fc-8d8a-66e9ebef2c82/payload-processing/0.log" Jul 24 20:12:02.245814 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:12:02.245780 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:12:32.034745 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:12:32.034711 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:12:56.100738 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:12:56.100706 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cxnzs_ecc30144-c524-45fc-8d8a-66e9ebef2c82/payload-processing/0.log" Jul 24 20:12:56.495931 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:12:56.495887 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:12:56.813339 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:12:56.813242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:12:57.696395 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:12:57.696361 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:00.229216 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:00.229178 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:00.326770 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:00.326736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:02.646057 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:02.646020 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:02.766072 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:02.766034 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:03.086376 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.086342 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:03.183977 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.183946 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:03.192695 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.192665 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622"] Jul 24 20:13:03.196324 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.196308 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.199433 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.199414 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-67ff20-dockercfg-8hlcr\"" Jul 24 20:13:03.199849 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.199833 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-67ff20\"" Jul 24 20:13:03.209150 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.209127 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622"] Jul 24 20:13:03.330164 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.330126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769a504-a010-42c3-82bf-5df3204684d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.330385 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.330268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4bp\" (UniqueName: \"kubernetes.io/projected/0769a504-a010-42c3-82bf-5df3204684d2-kube-api-access-8c4bp\") pod \"payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.431287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.431192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4bp\" (UniqueName: \"kubernetes.io/projected/0769a504-a010-42c3-82bf-5df3204684d2-kube-api-access-8c4bp\") pod \"payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.431287 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.431283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769a504-a010-42c3-82bf-5df3204684d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.432184 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.432156 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769a504-a010-42c3-82bf-5df3204684d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.439348 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.439323 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4bp\" (UniqueName: \"kubernetes.io/projected/0769a504-a010-42c3-82bf-5df3204684d2-kube-api-access-8c4bp\") pod \"payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.507788 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.507752 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:03.852425 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:03.852395 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622"] Jul 24 20:13:03.853919 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:13:03.853892 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0769a504_a010_42c3_82bf_5df3204684d2.slice/crio-cf15042b43a70692ec1e74bcd573d4beb331427e0140fd3bc649dd7997a8bbeb WatchSource:0}: Error finding container cf15042b43a70692ec1e74bcd573d4beb331427e0140fd3bc649dd7997a8bbeb: Status 404 returned error can't find the container with id cf15042b43a70692ec1e74bcd573d4beb331427e0140fd3bc649dd7997a8bbeb Jul 24 20:13:04.523566 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:04.523527 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" event={"ID":"0769a504-a010-42c3-82bf-5df3204684d2","Type":"ContainerStarted","Data":"921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e"} Jul 24 20:13:04.523832 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:04.523587 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" event={"ID":"0769a504-a010-42c3-82bf-5df3204684d2","Type":"ContainerStarted","Data":"cf15042b43a70692ec1e74bcd573d4beb331427e0140fd3bc649dd7997a8bbeb"} Jul 24 20:13:04.523832 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:04.523627 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:04.540223 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:04.540174 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" podStartSLOduration=1.54015941 podStartE2EDuration="1.54015941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:13:04.537755964 +0000 UTC m=+2814.633245170" watchObservedRunningTime="2026-07-24 20:13:04.54015941 +0000 UTC m=+2814.635648616" Jul 24 20:13:04.718793 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:04.718760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:11.206227 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:11.206189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:15.531448 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:15.531417 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:31.987741 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:31.987699 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:32.019592 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.019454 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622"] Jul 24 20:13:32.019838 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.019813 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" podUID="0769a504-a010-42c3-82bf-5df3204684d2" containerName="payload-pre-processing" containerID="cri-o://921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e" gracePeriod=30 Jul 24 20:13:32.284915 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.284889 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:32.299964 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.299934 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769a504-a010-42c3-82bf-5df3204684d2-plugins-config-volume\") pod \"0769a504-a010-42c3-82bf-5df3204684d2\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " Jul 24 20:13:32.300106 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.300066 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c4bp\" (UniqueName: \"kubernetes.io/projected/0769a504-a010-42c3-82bf-5df3204684d2-kube-api-access-8c4bp\") pod \"0769a504-a010-42c3-82bf-5df3204684d2\" (UID: \"0769a504-a010-42c3-82bf-5df3204684d2\") " Jul 24 20:13:32.300373 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.300334 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0769a504-a010-42c3-82bf-5df3204684d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0769a504-a010-42c3-82bf-5df3204684d2" (UID: "0769a504-a010-42c3-82bf-5df3204684d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 20:13:32.304305 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.304264 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0769a504-a010-42c3-82bf-5df3204684d2-kube-api-access-8c4bp" (OuterVolumeSpecName: "kube-api-access-8c4bp") pod "0769a504-a010-42c3-82bf-5df3204684d2" (UID: "0769a504-a010-42c3-82bf-5df3204684d2"). InnerVolumeSpecName "kube-api-access-8c4bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:13:32.401817 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.401783 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4bp\" (UniqueName: \"kubernetes.io/projected/0769a504-a010-42c3-82bf-5df3204684d2-kube-api-access-8c4bp\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:13:32.401817 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.401813 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769a504-a010-42c3-82bf-5df3204684d2-plugins-config-volume\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:13:32.680065 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.680031 2570 generic.go:356] "Generic (PLEG): container finished" podID="0769a504-a010-42c3-82bf-5df3204684d2" containerID="921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e" exitCode=0 Jul 24 20:13:32.680270 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.680079 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" Jul 24 20:13:32.680270 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.680117 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" event={"ID":"0769a504-a010-42c3-82bf-5df3204684d2","Type":"ContainerDied","Data":"921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e"} Jul 24 20:13:32.680270 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.680174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622" event={"ID":"0769a504-a010-42c3-82bf-5df3204684d2","Type":"ContainerDied","Data":"cf15042b43a70692ec1e74bcd573d4beb331427e0140fd3bc649dd7997a8bbeb"} Jul 24 20:13:32.680270 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.680197 2570 scope.go:117] "RemoveContainer" containerID="921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e" Jul 24 20:13:32.689652 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.689633 2570 scope.go:117] "RemoveContainer" containerID="921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e" Jul 24 20:13:32.689919 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:13:32.689897 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e\": container with ID starting with 921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e not found: ID does not exist" containerID="921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e" Jul 24 20:13:32.689998 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.689932 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e"} err="failed to get container status \"921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e\": rpc error: code = NotFound desc = could not find container \"921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e\": container with ID starting with 921a1a789f79b9a43124c728bbd1d5841cf24b649ff5ca2ae1ab14b7b22a7b1e not found: ID does not exist" Jul 24 20:13:32.696019 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.695980 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622"] Jul 24 20:13:32.698126 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:32.698103 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-67ff20-648776c5fb-9l622"] Jul 24 20:13:33.114446 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:33.114402 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:34.581484 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:34.581446 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0769a504-a010-42c3-82bf-5df3204684d2" path="/var/lib/kubelet/pods/0769a504-a010-42c3-82bf-5df3204684d2/volumes" Jul 24 20:13:35.881290 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:35.881247 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:36.288936 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:36.288904 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:42.312211 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:42.312179 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:42.412373 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:42.412334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:42.577060 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:42.576973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:13:42.813594 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:13:42.813546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:07.300192 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:07.300151 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:07.400605 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:07.400566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:07.452781 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:07.452747 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:11.343701 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:11.343670 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:11.744597 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:11.744554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:12.672264 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:12.672228 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:12.792096 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:12.792067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:16.767530 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:16.767491 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:52.922270 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:52.922236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:14:54.578695 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:14:54.578612 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:00.690386 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:00.690349 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:00.790762 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:00.790734 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:00.936479 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:00.936410 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79"] Jul 24 20:15:00.936736 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:00.936708 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" containerID="cri-o://55dbfb246400ae2ddff396a5ea485fc0874b3e1207a495bb352de8bc2d91a940" gracePeriod=30 Jul 24 20:15:00.971786 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:00.971714 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:01.189172 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:01.189121 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:03.160869 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:03.160830 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 20:15:03.161388 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:03.160890 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 20:15:06.199134 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.199066 2570 generic.go:356] "Generic (PLEG): container finished" podID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerID="55dbfb246400ae2ddff396a5ea485fc0874b3e1207a495bb352de8bc2d91a940" exitCode=0 Jul 24 20:15:06.199498 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.199134 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" event={"ID":"0b90c562-9ae4-4d34-a577-32ae904db2e3","Type":"ContainerDied","Data":"55dbfb246400ae2ddff396a5ea485fc0874b3e1207a495bb352de8bc2d91a940"} Jul 24 20:15:06.199498 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.199192 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" event={"ID":"0b90c562-9ae4-4d34-a577-32ae904db2e3","Type":"ContainerDied","Data":"f29d666705d391ed7401adf305844f04ec0664e9453c81e13837fb5e5ba50503"} Jul 24 20:15:06.199498 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.199208 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f29d666705d391ed7401adf305844f04ec0664e9453c81e13837fb5e5ba50503" Jul 24 20:15:06.199498 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.199222 2570 scope.go:117] "RemoveContainer" containerID="683d88897716d0ef89e6086515517a764988e24107c529d16d857cf3efe0ad38" Jul 24 20:15:06.205811 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.205791 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:15:06.293063 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293018 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrsbh\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-kube-api-access-zrsbh\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293104 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b90c562-9ae4-4d34-a577-32ae904db2e3-istiod-ca-cert\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293125 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-envoy\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293148 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-token\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293170 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-certs\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293197 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-socket\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293555 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293256 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-podinfo\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293555 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293281 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-data\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293555 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293311 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-credential-socket\") pod \"0b90c562-9ae4-4d34-a577-32ae904db2e3\" (UID: \"0b90c562-9ae4-4d34-a577-32ae904db2e3\") " Jul 24 20:15:06.293732 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293579 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:15:06.293732 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293646 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b90c562-9ae4-4d34-a577-32ae904db2e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 20:15:06.293843 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.293830 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:15:06.294145 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.294114 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:15:06.294494 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.294456 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:15:06.295640 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.295607 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 20:15:06.295742 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.295647 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:15:06.295903 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.295886 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-kube-api-access-zrsbh" (OuterVolumeSpecName: "kube-api-access-zrsbh") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "kube-api-access-zrsbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:15:06.295954 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.295895 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b90c562-9ae4-4d34-a577-32ae904db2e3" (UID: "0b90c562-9ae4-4d34-a577-32ae904db2e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:15:06.394632 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394581 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-podinfo\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394632 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394623 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-data\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394632 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394633 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-credential-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394632 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394643 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrsbh\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-kube-api-access-zrsbh\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394632 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394652 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b90c562-9ae4-4d34-a577-32ae904db2e3-istiod-ca-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394937 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394660 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-envoy\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394937 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394668 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b90c562-9ae4-4d34-a577-32ae904db2e3-istio-token\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394937 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394676 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-certs\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:06.394937 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:06.394684 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b90c562-9ae4-4d34-a577-32ae904db2e3-workload-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:15:07.205663 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:07.205633 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79" Jul 24 20:15:07.237422 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:07.237387 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79"] Jul 24 20:15:07.241092 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:07.241060 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-11ccc9-openshift-default-69c5f5d9f4-trw79"] Jul 24 20:15:08.582560 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:08.582528 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" path="/var/lib/kubelet/pods/0b90c562-9ae4-4d34-a577-32ae904db2e3/volumes" Jul 24 20:15:12.768861 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:12.768826 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:12.839832 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:12.839800 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:12.905853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:12.905814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:16.439536 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:16.439501 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:16.825992 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:16.825952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:22.625008 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:22.624969 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:22.725348 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:22.725318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:22.895451 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:22.895369 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:23.119270 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:23.119233 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:25.116887 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:25.116853 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:33.963532 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:33.963398 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:34.062781 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:34.062735 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:34.163333 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:34.163295 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:38.425067 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:38.425025 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:38.803849 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:38.803816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:44.936096 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:44.936058 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:45.036423 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:45.036387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:45.204706 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:45.204624 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:45.439062 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:45.439027 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:54.865409 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:54.865360 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:56.615026 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:56.614995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:56.694143 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:56.694115 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:15:56.754383 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:15:56.754354 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:00.623982 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:00.623951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:01.023491 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:01.023441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:06.823555 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:06.823512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:06.884909 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:06.884875 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:07.070211 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:07.070181 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m"] Jul 24 20:16:07.070440 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:07.070416 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" containerID="cri-o://cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0" gracePeriod=30 Jul 24 20:16:07.093048 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:07.092915 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:07.316434 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:07.316394 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:10.662285 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:10.662249 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:16:10.666239 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:10.666210 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:16:12.327228 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.327199 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:16:12.409197 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409109 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-podinfo\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409197 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409156 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-token\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409197 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409194 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-data\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409509 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409252 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istiod-ca-cert\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409509 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409367 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-kube-api-access-h7djd\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409509 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409436 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-certs\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409509 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409495 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-socket\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409734 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409536 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-envoy\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409734 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409578 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-credential-socket\") pod \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\" (UID: \"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5\") " Jul 24 20:16:12.409841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409737 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:16:12.409841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409744 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:16:12.409841 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409746 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 20:16:12.410007 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.409951 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:16:12.410058 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.410034 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:16:12.410115 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.410098 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istiod-ca-cert\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.410155 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.410119 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-certs\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.410155 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.410132 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-workload-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.410155 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.410145 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-credential-socket\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.411799 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.411769 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 20:16:12.411918 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.411793 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 20:16:12.411978 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.411967 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:16:12.412018 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.411988 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-kube-api-access-h7djd" (OuterVolumeSpecName: "kube-api-access-h7djd") pod "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" (UID: "e6d08df4-ff68-4ed9-ae37-aef44bdc67a5"). InnerVolumeSpecName "kube-api-access-h7djd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 20:16:12.511224 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.511198 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-envoy\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.511224 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.511219 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-podinfo\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.511224 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.511227 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-token\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.511425 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.511237 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-istio-data\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.511425 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.511245 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5-kube-api-access-h7djd\") on node \"ip-10-0-141-238.ec2.internal\" DevicePath \"\"" Jul 24 20:16:12.563373 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.563338 2570 generic.go:356] "Generic (PLEG): container finished" podID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerID="cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0" exitCode=0 Jul 24 20:16:12.563526 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.563382 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" Jul 24 20:16:12.563526 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.563412 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" event={"ID":"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5","Type":"ContainerDied","Data":"cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0"} Jul 24 20:16:12.563526 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.563497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m" event={"ID":"e6d08df4-ff68-4ed9-ae37-aef44bdc67a5","Type":"ContainerDied","Data":"84b9aef1412e5ce674ae8caf63b08c7fea1feb3f5a714147368a16c0402b427e"} Jul 24 20:16:12.563526 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.563522 2570 scope.go:117] "RemoveContainer" containerID="cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0" Jul 24 20:16:12.573561 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.573540 2570 scope.go:117] "RemoveContainer" containerID="7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb" Jul 24 20:16:12.582037 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.582020 2570 scope.go:117] "RemoveContainer" containerID="cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0" Jul 24 20:16:12.582274 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:16:12.582256 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0\": container with ID starting with cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0 not found: ID does not exist" containerID="cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0" Jul 24 20:16:12.582335 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.582283 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0"} err="failed to get container status \"cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0\": rpc error: code = NotFound desc = could not find container \"cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0\": container with ID starting with cd19358c85294fbe35fb32b519107e711ed1b563b14278cfe48dc3340d2557a0 not found: ID does not exist" Jul 24 20:16:12.582335 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.582301 2570 scope.go:117] "RemoveContainer" containerID="7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb" Jul 24 20:16:12.582530 ip-10-0-141-238 kubenswrapper[2570]: E0724 20:16:12.582514 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb\": container with ID starting with 7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb not found: ID does not exist" containerID="7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb" Jul 24 20:16:12.582583 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.582534 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb"} err="failed to get container status \"7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb\": rpc error: code = NotFound desc = could not find container \"7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb\": container with ID starting with 7c20d8091663df7b4678da732e57e76c788d4620a8beef45073c69de805ce8bb not found: ID does not exist" Jul 24 20:16:12.607185 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.607158 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m"] Jul 24 20:16:12.611691 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:12.611670 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-3e15b4-openshift-default-65f976bbd8-s2w6m"] Jul 24 20:16:13.149482 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:13.149437 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vgqt2_54717a5e-11ed-46fb-98a4-a7613ba540d5/manager/0.log" Jul 24 20:16:13.266127 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:13.266100 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t6tch_7ef36049-ec6f-4beb-9e2f-fbee5edb3920/manager/0.log" Jul 24 20:16:13.507628 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:13.507605 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r82sc_1cd435c2-ab14-43f0-8728-1f64a2b193b3/server/0.log" Jul 24 20:16:13.633275 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:13.633245 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j7z9v_ab251975-fd07-4a97-9093-a4d55deca6c0/manager/0.log" Jul 24 20:16:14.013542 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:14.013521 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69f558d7f4-dr6wk_b89458d2-31af-4662-93db-0378273ff84f/copy-manifests/0.log" Jul 24 20:16:14.026031 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:14.026009 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69f558d7f4-dr6wk_b89458d2-31af-4662-93db-0378273ff84f/manager/0.log" Jul 24 20:16:14.372549 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:14.372476 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6896c585f8-sfgcf_d3c30409-6ca8-4775-864f-b8f08ea55005/maas-api/0.log" Jul 24 20:16:14.580888 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:14.580854 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" path="/var/lib/kubelet/pods/e6d08df4-ff68-4ed9-ae37-aef44bdc67a5/volumes" Jul 24 20:16:14.822978 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:14.822948 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hm2vx_bb9b9575-1402-4b68-928f-81862b667417/postgres/0.log" Jul 24 20:16:16.342978 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:16.342953 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-6kzsp_f234cac5-12df-4873-b0b4-245d6bcdabe8/manager/0.log" Jul 24 20:16:16.456494 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:16.456448 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-96jc5_f374077a-a01c-4188-a847-e9bafc7d5d5a/kuadrant-console-plugin/0.log" Jul 24 20:16:16.569480 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:16.569443 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-g5j2r_c10c45e4-b62e-4a06-a0f2-42e3f58d9720/registry-server/0.log" Jul 24 20:16:16.918587 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:16.918553 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-56mj7_03c79a14-1772-4ea4-8013-dea36ac2449a/manager/0.log" Jul 24 20:16:17.274304 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:17.274276 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s_d2b07f51-050d-4875-859b-1d627ac8a572/istio-proxy/0.log" Jul 24 20:16:17.625030 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:17.624962 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cff79d5b6-t5f5h_be6fb9cb-80af-4da4-be5f-a7a4089505f8/kube-auth-proxy/0.log" Jul 24 20:16:17.743206 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:17.743175 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hcz6h_375d0858-f3b5-4c3c-9003-181bb349d96a/istio-proxy/1.log" Jul 24 20:16:17.864169 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:17.864143 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8fgfw_fc9d3da8-dc69-4f32-bf97-762cf5140d3a/payload-pre-processing/0.log" Jul 24 20:16:17.983538 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:17.983462 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cxnzs_ecc30144-c524-45fc-8d8a-66e9ebef2c82/payload-processing/0.log" Jul 24 20:16:18.100675 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:18.100644 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56947db777-2bfh7_9996735c-0afe-414c-b463-b2f4c223d41c/router/0.log" Jul 24 20:16:18.558083 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:18.558057 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lrtbq_2787d13f-2b4d-4659-a627-67b8ecfba54e/main/0.log" Jul 24 20:16:18.668684 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:18.668656 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-fkddp_caaa698c-1694-4088-a815-049aa4f3a707/main/0.log" Jul 24 20:16:24.060825 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:24.060790 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:24.120923 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:24.120890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:26.877183 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:26.877150 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-plptd_8132dea3-f97b-44d0-b452-bed78f859762/global-pull-secret-syncer/0.log" Jul 24 20:16:26.928389 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:26.928360 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" Jul 24 20:16:27.065238 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:27.065207 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-238.ec2.internal_2a7ab8408f0b62fc7e97aef1336f6df8/haproxy/0.log" Jul 24 20:16:32.468752 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:32.468723 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-6kzsp_f234cac5-12df-4873-b0b4-245d6bcdabe8/manager/0.log" Jul 24 20:16:32.497578 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:32.497556 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-96jc5_f374077a-a01c-4188-a847-e9bafc7d5d5a/kuadrant-console-plugin/0.log" Jul 24 20:16:32.543844 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:32.543816 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-g5j2r_c10c45e4-b62e-4a06-a0f2-42e3f58d9720/registry-server/0.log" Jul 24 20:16:32.837139 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:32.837092 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-56mj7_03c79a14-1772-4ea4-8013-dea36ac2449a/manager/0.log" Jul 24 20:16:34.694658 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:34.694570 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-5xb5f_7267d569-0982-4797-af25-32d1752be10b/cluster-monitoring-operator/0.log" Jul 24 20:16:34.845440 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:34.845406 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6c8k9_ed45484f-b97c-4364-9c25-39daa56d9f97/node-exporter/0.log" Jul 24 20:16:34.867746 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:34.867723 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6c8k9_ed45484f-b97c-4364-9c25-39daa56d9f97/kube-rbac-proxy/0.log" Jul 24 20:16:34.895882 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:34.895859 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6c8k9_ed45484f-b97c-4364-9c25-39daa56d9f97/init-textfile/0.log" Jul 24 20:16:36.718397 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.718365 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-99pmx_c246e701-32dc-4ec9-a53d-b4e9cffdff6b/networking-console-plugin/0.log" Jul 24 20:16:36.729237 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729200 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb"] Jul 24 20:16:36.729801 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729783 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" Jul 24 20:16:36.729853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729804 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" Jul 24 20:16:36.729853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729813 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" Jul 24 20:16:36.729853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729818 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" Jul 24 20:16:36.729853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729828 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0769a504-a010-42c3-82bf-5df3204684d2" containerName="payload-pre-processing" Jul 24 20:16:36.729853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729833 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0769a504-a010-42c3-82bf-5df3204684d2" containerName="payload-pre-processing" Jul 24 20:16:36.729853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729847 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" Jul 24 20:16:36.729853 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729853 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" Jul 24 20:16:36.730097 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729927 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6d08df4-ff68-4ed9-ae37-aef44bdc67a5" containerName="istio-proxy" Jul 24 20:16:36.730097 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729937 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" Jul 24 20:16:36.730097 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729943 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0769a504-a010-42c3-82bf-5df3204684d2" containerName="payload-pre-processing" Jul 24 20:16:36.730097 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.729950 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b90c562-9ae4-4d34-a577-32ae904db2e3" containerName="istio-proxy" Jul 24 20:16:36.733281 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.733260 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.735688 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.735670 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g4kbv\"/\"default-dockercfg-d5j6l\"" Jul 24 20:16:36.735779 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.735671 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4kbv\"/\"openshift-service-ca.crt\"" Jul 24 20:16:36.736668 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.736654 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4kbv\"/\"kube-root-ca.crt\"" Jul 24 20:16:36.744338 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.744313 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb"] Jul 24 20:16:36.766962 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.766926 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:36.824311 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.824274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tw8sp_4048e8da-ece3-4dcb-ae2d-f737edc52db7/konnectivity-agent/0.log" limit=1024 Jul 24 20:16:36.831083 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.831051 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-proc\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.831223 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.831093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcx9n\" (UniqueName: \"kubernetes.io/projected/c0046c9a-8ace-4082-a3fa-c834cd80e545-kube-api-access-kcx9n\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.831223 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.831151 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-lib-modules\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.831223 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.831195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-podres\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.831335 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.831301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-sys\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932001 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.931962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-sys\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932001 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-proc\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932280 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcx9n\" (UniqueName: \"kubernetes.io/projected/c0046c9a-8ace-4082-a3fa-c834cd80e545-kube-api-access-kcx9n\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932280 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-lib-modules\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932280 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-podres\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932280 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932095 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-sys\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932280 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932095 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-proc\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932280 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932220 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-podres\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.932280 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.932235 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0046c9a-8ace-4082-a3fa-c834cd80e545-lib-modules\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:36.940417 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:36.940380 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcx9n\" (UniqueName: \"kubernetes.io/projected/c0046c9a-8ace-4082-a3fa-c834cd80e545-kube-api-access-kcx9n\") pod \"perf-node-gather-daemonset-t5dpb\" (UID: \"c0046c9a-8ace-4082-a3fa-c834cd80e545\") " pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:37.043677 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.043643 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:37.186631 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.186604 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb"] Jul 24 20:16:37.189477 ip-10-0-141-238 kubenswrapper[2570]: W0724 20:16:37.189442 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc0046c9a_8ace_4082_a3fa_c834cd80e545.slice/crio-8d3ade507ef675f21d1dc3499a7544248b72a29e577de660be044c58a2162a78 WatchSource:0}: Error finding container 8d3ade507ef675f21d1dc3499a7544248b72a29e577de660be044c58a2162a78: Status 404 returned error can't find the container with id 8d3ade507ef675f21d1dc3499a7544248b72a29e577de660be044c58a2162a78 Jul 24 20:16:37.432992 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.432904 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/1.log" Jul 24 20:16:37.437654 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.437629 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-frwvb_de633ac5-2c17-401d-8529-c02d480d9f25/console-operator/2.log" Jul 24 20:16:37.709569 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.709444 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" event={"ID":"c0046c9a-8ace-4082-a3fa-c834cd80e545","Type":"ContainerStarted","Data":"5717d0449c4a0e686baaa6a980d0c1db46a97ee9c7ae91e6184c4adb8e5630e0"} Jul 24 20:16:37.709569 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.709530 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" event={"ID":"c0046c9a-8ace-4082-a3fa-c834cd80e545","Type":"ContainerStarted","Data":"8d3ade507ef675f21d1dc3499a7544248b72a29e577de660be044c58a2162a78"} Jul 24 20:16:37.709569 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.709559 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:37.725976 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.725922 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" podStartSLOduration=1.7259046329999999 podStartE2EDuration="1.725904633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:37.72460796 +0000 UTC m=+3027.820097167" watchObservedRunningTime="2026-07-24 20:16:37.725904633 +0000 UTC m=+3027.821393839" Jul 24 20:16:37.895625 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.895588 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fd4b7575f-ghkjw_00615fdb-ab53-46f0-9aba-4ecc4c9ccc00/console/0.log" Jul 24 20:16:37.926320 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:37.926288 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-xrzx2_d0267eb1-f96c-4d53-8651-438bb2fbcd4e/download-server/0.log" Jul 24 20:16:38.476691 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:38.476660 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-lgkl2_d1cdec05-18f6-4d64-ac29-65b77e5ff36b/volume-data-source-validator/0.log" Jul 24 20:16:39.520444 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:39.520410 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ghxhc_938f432d-fa76-41c9-b82b-25f85fbb30be/dns/0.log" Jul 24 20:16:39.541811 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:39.541787 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ghxhc_938f432d-fa76-41c9-b82b-25f85fbb30be/kube-rbac-proxy/0.log" Jul 24 20:16:39.631650 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:39.631618 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j425q_54646590-a397-4aab-8b6f-a4de53ca338d/dns-node-resolver/0.log" Jul 24 20:16:40.254255 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:40.254222 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqltr_4a2b5609-a8ac-4be7-a8d6-91bce606bfd1/node-ca/0.log" Jul 24 20:16:41.176114 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:41.176084 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s_d2b07f51-050d-4875-859b-1d627ac8a572/istio-proxy/0.log" Jul 24 20:16:41.363855 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:41.363823 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cff79d5b6-t5f5h_be6fb9cb-80af-4da4-be5f-a7a4089505f8/kube-auth-proxy/0.log" Jul 24 20:16:41.389515 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:41.389463 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hcz6h_375d0858-f3b5-4c3c-9003-181bb349d96a/istio-proxy/0.log" Jul 24 20:16:41.415426 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:41.415397 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hcz6h_375d0858-f3b5-4c3c-9003-181bb349d96a/istio-proxy/1.log" Jul 24 20:16:41.452229 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:41.452154 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8fgfw_fc9d3da8-dc69-4f32-bf97-762cf5140d3a/payload-pre-processing/0.log" Jul 24 20:16:41.481988 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:41.481961 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cxnzs_ecc30144-c524-45fc-8d8a-66e9ebef2c82/payload-processing/0.log" Jul 24 20:16:41.519434 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:41.519401 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56947db777-2bfh7_9996735c-0afe-414c-b463-b2f4c223d41c/router/0.log" Jul 24 20:16:42.320895 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:42.320865 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mnvj7_028f41d9-d9c3-44ed-85af-4c7078cf2cd4/serve-healthcheck-canary/0.log" Jul 24 20:16:42.817781 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:42.817748 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-s6q2j_faeb92d0-d3b8-4602-95ce-941edc7337dd/insights-operator/0.log" Jul 24 20:16:42.818458 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:42.818441 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-s6q2j_faeb92d0-d3b8-4602-95ce-941edc7337dd/insights-operator/1.log" Jul 24 20:16:42.838655 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:42.838627 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h59ph_d30c97b9-f517-4a45-896d-5cdf53128137/kube-rbac-proxy/0.log" Jul 24 20:16:42.858084 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:42.858055 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h59ph_d30c97b9-f517-4a45-896d-5cdf53128137/exporter/0.log" Jul 24 20:16:42.879127 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:42.879103 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h59ph_d30c97b9-f517-4a45-896d-5cdf53128137/extractor/0.log" Jul 24 20:16:43.725379 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:43.725348 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g4kbv/perf-node-gather-daemonset-t5dpb" Jul 24 20:16:45.114522 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:45.114494 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vgqt2_54717a5e-11ed-46fb-98a4-a7613ba540d5/manager/0.log" Jul 24 20:16:45.318537 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:45.318502 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t6tch_7ef36049-ec6f-4beb-9e2f-fbee5edb3920/manager/0.log" Jul 24 20:16:45.647186 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:45.647162 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r82sc_1cd435c2-ab14-43f0-8728-1f64a2b193b3/server/0.log" Jul 24 20:16:45.737744 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:45.737715 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j7z9v_ab251975-fd07-4a97-9093-a4d55deca6c0/manager/0.log" Jul 24 20:16:45.924667 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:45.924634 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69f558d7f4-dr6wk_b89458d2-31af-4662-93db-0378273ff84f/manager/0.log" Jul 24 20:16:45.943735 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:45.943712 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69f558d7f4-dr6wk_b89458d2-31af-4662-93db-0378273ff84f/copy-manifests/0.log" Jul 24 20:16:47.902714 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:47.902673 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s_d2b07f51-050d-4875-859b-1d627ac8a572/istio-proxy/0.log" Jul 24 20:16:48.070967 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:48.070938 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cff79d5b6-t5f5h_be6fb9cb-80af-4da4-be5f-a7a4089505f8/kube-auth-proxy/0.log" Jul 24 20:16:48.088705 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:48.088630 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hcz6h_375d0858-f3b5-4c3c-9003-181bb349d96a/istio-proxy/0.log" Jul 24 20:16:48.114592 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:48.114558 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hcz6h_375d0858-f3b5-4c3c-9003-181bb349d96a/istio-proxy/1.log" Jul 24 20:16:48.134277 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:48.134246 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8fgfw_fc9d3da8-dc69-4f32-bf97-762cf5140d3a/payload-pre-processing/0.log" Jul 24 20:16:48.155249 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:48.155217 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cxnzs_ecc30144-c524-45fc-8d8a-66e9ebef2c82/payload-processing/0.log" Jul 24 20:16:48.167671 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:48.167645 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56947db777-2bfh7_9996735c-0afe-414c-b463-b2f4c223d41c/router/0.log" Jul 24 20:16:49.510556 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:49.510529 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x6p8s_d2b07f51-050d-4875-859b-1d627ac8a572/istio-proxy/0.log" Jul 24 20:16:49.682214 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:49.682183 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cff79d5b6-t5f5h_be6fb9cb-80af-4da4-be5f-a7a4089505f8/kube-auth-proxy/0.log" Jul 24 20:16:49.699824 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:49.699795 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hcz6h_375d0858-f3b5-4c3c-9003-181bb349d96a/istio-proxy/0.log" Jul 24 20:16:49.725489 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:49.725439 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hcz6h_375d0858-f3b5-4c3c-9003-181bb349d96a/istio-proxy/1.log" Jul 24 20:16:49.750845 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:49.750822 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8fgfw_fc9d3da8-dc69-4f32-bf97-762cf5140d3a/payload-pre-processing/0.log" Jul 24 20:16:49.771042 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:49.770986 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cxnzs_ecc30144-c524-45fc-8d8a-66e9ebef2c82/payload-processing/0.log" Jul 24 20:16:49.785480 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:49.785443 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56947db777-2bfh7_9996735c-0afe-414c-b463-b2f4c223d41c/router/0.log" Jul 24 20:16:52.466964 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:52.466929 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9txg6_fb7155bf-ad1d-42c4-b91d-b036ef58de8f/migrator/0.log" Jul 24 20:16:52.486788 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:52.486761 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9txg6_fb7155bf-ad1d-42c4-b91d-b036ef58de8f/graceful-termination/0.log" Jul 24 20:16:52.855674 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:52.855646 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2_fc62b1cf-7e3a-4df4-9272-82d6e407dfec/kube-storage-version-migrator-operator/1.log" Jul 24 20:16:52.856669 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:52.856648 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-gqhs2_fc62b1cf-7e3a-4df4-9272-82d6e407dfec/kube-storage-version-migrator-operator/0.log" Jul 24 20:16:54.021790 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.021762 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cx256_3e4d9366-97b2-4601-89c8-04e78cccb813/kube-multus-additional-cni-plugins/0.log" Jul 24 20:16:54.044386 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.044357 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cx256_3e4d9366-97b2-4601-89c8-04e78cccb813/egress-router-binary-copy/0.log" Jul 24 20:16:54.080248 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.080216 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cx256_3e4d9366-97b2-4601-89c8-04e78cccb813/cni-plugins/0.log" Jul 24 20:16:54.099671 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.099644 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cx256_3e4d9366-97b2-4601-89c8-04e78cccb813/bond-cni-plugin/0.log" Jul 24 20:16:54.118452 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.118425 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cx256_3e4d9366-97b2-4601-89c8-04e78cccb813/routeoverride-cni/0.log" Jul 24 20:16:54.137914 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.137886 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cx256_3e4d9366-97b2-4601-89c8-04e78cccb813/whereabouts-cni-bincopy/0.log" Jul 24 20:16:54.158023 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.157996 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cx256_3e4d9366-97b2-4601-89c8-04e78cccb813/whereabouts-cni/0.log" Jul 24 20:16:54.365409 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.365332 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gzggx_9c403e81-e95d-4a0b-80f5-db75cb24f9c4/kube-multus/0.log" Jul 24 20:16:54.385659 ip-10-0-141-238 kubenswrapper[2570]: I0724 20:16:54.385628 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6l22f_06dc9231-d8aa-465a-aa18-a1e9e46cbd18/network-metrics-daemon/0.log"